Imaging device

ABSTRACT

An imaging element comprising: an imaging unit that has: a plurality of groups each including at least one pixel; and a plurality of signal readout units that are each provided to each of the groups and read out a signal from the pixel; and a control unit that controls the signal readout unit in at least one group among the plurality of groups is provided. Each of the plurality of groups may include a plurality of the pixels. The control unit may select at least one group among the plurality of groups and control the signal readout unit by using a control parameter that is different from a control parameter that is used for another group among the plurality of groups.

CROSS REFERENCE TO RELATED APPLICATION

The contents of the following Japanese and International patentapplications are incorporated herein by reference:

2012-105316 filed on May 2, 2012,

2012-139026 filed on Jun. 20, 2012,

2012-142126 filed on Jun. 25, 2012,

2012-149844 filed on Jul. 3, 2012,

2012-149946 filed on Jul. 3, 2012, and

PCT/JP2013/002927 filed on May 2, 2013

BACKGROUND

1. Technical Field

The present invention relates to an imaging device.

2. Related Art

An imaging unit in which a backside illuminating type imaging chip and asignal processing chip are connected, via microbumps, for each cell unitincluding a plurality of pixels is known.

PRIOR ART DOCUMENTS Patent Literatures

[Patent Literature 1] Japanese Patent Application Publication No.2006-49361

SUMMARY

There is a control line for each cell in the imaging unit. However, anelectrical charge accumulation period and readout of pixel signals arenot minutely controlled on a cell-by-cell basis.

A first aspect of the present invention provides an imaging elementcomprising: an imaging unit that has: a plurality of groups eachincluding at least one pixel; and a plurality of signal readout unitsthat are each provided to each of the groups and read out a signal fromthe pixel; and a control unit that controls the signal readout unit inat least one group among the plurality of groups.

A second aspect of the present invention provides an imaging elementcomprising: an imaging unit that has: a plurality of groups eachincluding at least one pixel; and a plurality of signal readout unitsthat are each provided to each of the groups and read out a signal fromthe pixel; and a plurality of control units that are each provided toeach of the groups, and controls the signal readout unit based on asignal from the pixel.

A third aspect of the present invention provides an imaging elementcomprising: an imaging unit having an imaging area in which a firstpixel and a second pixel are provided, a first readout circuit thatreads out a first pixel signal output from the first pixel, and a secondreadout circuit that reads out a second pixel signal output from thesecond pixel; a first computing unit that computes a first evaluationvalue based on the first pixel signal; a second computing unit thatcomputes a second evaluation value based on the second pixel signal; afirst control unit that performs control on exposure or readout of thefirst pixel based on the first evaluation value; and a second controlunit that performs control on exposure or readout of the second pixelbased on the second evaluation value.

A fourth aspect of the present invention provides an imaging elementcomprising: an imaging unit that has: a plurality of groups eachincluding at least one pixel; and a plurality of signal readout unitsthat are each provided to each of the groups and read out a signal fromthe pixel; and a plurality of computing units that are provided to eachof the groups and transmit information about control on the signalreadout unit to an image processing unit that performs image processingon the signal.

A fifth aspect of the present invention provides an imaging elementcomprising: an imaging unit having an imaging area in which a firstpixel and a second pixel are disposed, a first readout circuit thatreads out a first pixel signal output from the first pixel, and a secondreadout circuit that reads out a second pixel signal output from thesecond pixel; a first computing unit that computes a first evaluationvalue based on the first pixel signal, and transmits the computed firstevaluation value to an image processing unit in a subsequent step thatperforms image processing on first pixel data that corresponds to thefirst pixel signal; and a second computing unit that computes a secondevaluation value based on the second pixel signal, and transmits thecomputed second evaluation value to an image processing unit in asubsequent step that performs image processing on second pixel data thatcorresponds to the second pixel signal.

A sixth aspect of the present invention provides an imaging elementcomprising: an imaging unit that has a plurality of groups eachincluding at least one pixel; and a storage unit that has a plurality ofstorage blocks that are provided corresponding to the plurality ofgroups, and store a signal from a pixel in the respectivelycorresponding group, and store a signal from a pixel in a group otherthan the respectively corresponding group.

The summary clause does not necessarily describe all necessary featuresof the embodiments of the present invention. The present invention mayalso be a sub-combination of the features described above.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a sectional view of a backside illuminating type MOS imagingelement according to the present embodiment.

FIG. 2 is a diagram for explaining a pixel array and a unit group of theimaging chip.

FIG. 3 is a schematic that corresponds to a unit group of the imagingchip.

FIG. 4 is a block diagram showing a functional configuration of animaging element.

FIG. 5 is a block diagram showing a configuration of an imaging deviceaccording to the present embodiment.

FIG. 6 is a functional block diagram of the image processing unit.

FIG. 7 is a flowchart that illustrates operations of an imaging deviceto generate and record a motion image.

FIG. 8 illustrates one example of an image imaged by an imaging element.

FIG. 9 illustrates one example of an image imaged by an imaging element.

FIG. 10 illustrates a relationship between respective frame rates andoutput timing of image signals.

FIG. 11 schematically illustrates an attention area motion image and aperipheral area motion image generated by the motion image generatingunit.

FIG. 12 illustrates one example of the header information added by themotion image generating unit.

FIG. 13 is a flowchart that illustrates operations of an imaging deviceto reproduce and display a motion image.

FIG. 14 is a flowchart that illustrates another example of operations ofthe imaging device to generate and record a motion image.

FIG. 15 illustrates an example of pixels to be read out at the thinningrate of 0.5 in one unit group.

FIG. 16 is a flowchart that illustrates operations of an imaging deviceto reproduce and display a motion image.

FIG. 17 is a diagram for explaining an example of a scene and areadivision.

FIG. 18 is a diagram for explaining electrical charge accumulationcontrol for the respective areas divided in the example in FIG. 17.

FIG. 19 is a table that indicates a relationship between the number oftimes of integration and the dynamic range.

FIG. 20 is a flow diagram showing processing of imaging operations.

FIG. 21 is a block diagram that illustrates a specific configuration ofthe signal processing chip as one example.

FIG. 22 is a sectional view of another backside illuminating type MOSimaging element according to the present embodiment.

FIG. 23 is a diagram for explaining a pixel array and a unit group ofthe imaging chip.

FIG. 24 is a schematic that corresponds to a unit group of the imagingchip.

FIG. 25 is a block diagram showing a configuration of an imaging deviceaccording to the present embodiment.

FIG. 26 is a block diagram that illustrates a specific configuration ofthe signal processing chip as one example.

FIG. 27 shows one example of functional blocks of the arithmetic circuit1415.

FIG. 28 illustrates one example of correspondence between inter-framedifferences d, and frame rates f.

FIG. 29 illustrates one example of an image imaged by an imagingelement.

FIG. 30 illustrates one example of an image imaged by an imagingelement.

FIG. 31 shows one example of functional blocks of another arithmeticcircuit.

FIG. 32 illustrates an example of pixels 1188 to be read out at thethinning rate of 0.5 in one unit group.

FIG. 33 illustrates one example of functional blocks of still anotherarithmetic circuit.

FIG. 34 schematically illustrates a relationship between gains and pixelsignals.

FIG. 35 is a sectional view of a backside illuminating type MOS imagingelement according to the present embodiment.

FIG. 36 is a diagram for explaining a pixel array and a pixel block ofthe imaging chip.

FIG. 37 is a schematic that corresponds to the pixel block of theimaging chip.

FIG. 38 is a diagram that illustrates a part of a configuration of animaging element, and its operation example.

FIG. 39 is a block diagram showing a configuration of an imaging deviceaccording to the present embodiment.

FIG. 40 is a functional block diagram of the image processing unit.

FIG. 41 is a flowchart that illustrates operations of an imaging deviceto generate and record a motion image.

FIG. 42 illustrates one example of an image imaged by an imagingelement.

FIG. 43 illustrates one example of an image imaged by an imagingelement.

FIG. 44 illustrates a relationship between respective frame rates andoutput timing of image signals.

FIG. 45 schematically illustrates an attention area motion image and aperipheral area motion image generated by the motion image generatingunit.

FIG. 46 illustrates one example of the header information added by themotion image generating unit.

FIG. 47 is a flowchart that illustrates operations of an imaging deviceto reproduce and display a motion image.

FIG. 48 is a flowchart that illustrates another example of operations ofthe imaging device to generate and record a motion image.

FIG. 49 illustrates an example of pixels to be read out at the thinningrate of 0.5 in one pixel block.

FIG. 50 is a flowchart that illustrates operations of an imaging deviceto reproduce and display a motion image.

FIG. 51A is a diagram for explaining a scene.

FIG. 51B is a diagram for explaining area division.

FIG. 52 is a diagram for explaining electrical charge accumulationcontrol for the respective areas divided in the example in FIG. 51B.

FIG. 53 is a table that indicates a relationship between the number oftimes of integration and the dynamic range.

FIG. 54 is a flow diagram showing processing of imaging operations.

FIG. 55 is a block diagram that illustrates a specific configuration ofthe signal processing chip as one example.

FIG. 56 is a block diagram showing a configuration of a peripheral pixeldata processing unit.

FIG. 57 is a block diagram that illustrates one example of aconfiguration of an arithmetic circuit.

FIG. 58 is a flowchart that illustrates an example of operations of anarithmetic circuit.

FIG. 59 illustrates a configuration of a data array generated by anoutput circuit.

FIG. 60 illustrates contents of the data array illustrated in FIG. 59.

FIG. 61 is a sectional view of a backside illuminating type MOS imagingelement according to the present embodiment.

FIG. 62 is a diagram for explaining a pixel array and a pixel block ofthe imaging chip.

FIG. 63 is a schematic that corresponds to the pixel block of theimaging chip.

FIG. 64A is a diagram that illustrates a part of a configuration of animaging element, and its operation example.

FIG. 64B is a diagram that illustrates another operation example of theimaging element.

FIG. 64C is a diagram that illustrates another operation example of theimaging element.

FIG. 65 is a block diagram showing a configuration of an imaging deviceaccording to the present embodiment.

FIG. 66 is a functional block diagram of the image processing unit.

FIG. 67 is a flowchart that illustrates operations of an imaging deviceto generate and record a motion image.

FIG. 68 illustrates one example of an image imaged by an imagingelement.

FIG. 69 illustrates one example of an image imaged by an imagingelement.

FIG. 70 illustrates a relationship between respective frame rates andoutput timing of image signals.

FIG. 71 schematically illustrates an attention area motion image and aperipheral area motion image generated by the motion image generatingunit.

FIG. 72 illustrates one example of the header information added by themotion image generating unit.

FIG. 73 is a flowchart that illustrates operations of an imaging deviceto reproduce and display a motion image.

FIG. 74 is a plan view of a pixel area of an imaging element and itsoperation example.

FIG. 75 is a plan view of another configuration of a pixel area of animaging element and its operation example.

FIG. 76 is a plan view of another configuration of a pixel area of animaging element and its operation example.

FIG. 77 is a plan view of another configuration of a pixel area of animaging element and its operation example.

FIG. 78 is a plan view of another configuration of a pixel area of animaging element and its operation example.

FIG. 79 is a flowchart that illustrates another example of operations ofthe imaging device to generate and record a motion image.

FIG. 80 illustrates an example of pixels to be read out at the thinningrate of 0.5.

FIG. 81 is a flowchart that illustrates operations of an imaging deviceto reproduce and display a motion image.

FIG. 82A is a diagram for explaining a scene.

FIG. 82B is a diagram for explaining area division.

FIG. 83 is a diagram for explaining electrical charge accumulationcontrol for the respective areas divided in the example in FIG. 82B.

FIG. 84 is a table that indicates a relationship between the number oftimes of integration and the dynamic range.

FIG. 85 is a flow diagram showing processing of imaging operations.

FIG. 86 is a block diagram that illustrates a specific configuration ofthe signal processing chip as one example.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Hereinafter, (some) embodiment(s) of the present invention will bedescribed. The embodiment(s) do(es) not limit the invention according tothe claims, and all the combinations of the features described in theembodiment(s) are not necessarily essential to means provided by aspectsof the invention.

FIG. 1 is a sectional view of a backside illuminating type imagingelement 100 according to the present embodiment. The imaging element 100includes an imaging chip 113 that outputs a pixel signal correspondingto incident light, a signal processing chip 111 that processes the pixelsignal, and a memory chip 112 that stores the pixel signal. Theseimaging chip 113, signal processing chip 111, and memory chip 112 arelayered, and are electrically connected with each other via conductivebumps 109, such as Cu.

Note that, as illustrated, incident light is incident mainly in the Zaxis positive direction that is indicated with an outlined arrow. In thepresent embodiment, the surface of the imaging chip 113 on a side onwhich the incident light is incident is called a backside. Also, asindicated with coordinate axes, the leftward direction on the figurethat is orthogonal to the Z axis is referred to as the X axis positivedirection, and the front side direction in the figure that is orthogonalto the Z and X axes is referred to as the Y axis positive direction. Inseveral figures mentioned below, the coordinate axes are displayed suchthat the orientation of each figure can be known on the basis of thecoordinate axes in FIG. 1.

One example of the imaging chip 113 is a backside illuminating type MOSimage sensor. A PD layer 106 is disposed on a backside of aninterconnection layer 108. The PD layer 106 has a plurality of PDs(photo diodes) 104 that are two-dimensionally disposed and accumulateelectrical charges according to incident light, and transistors 105provided corresponding to the PDs 104.

Color filters 102 are provided on the incident light incidence side ofthe PD layer 106 via a passivation film 103. There is a plurality oftypes of the color filters 102 that allow passage of mutually differentwavelength ranges, and the color filters 102 are arrayed particularlycorresponding to the respective PDs 104. The arrays of the color filters102 are described below. A set of the color filter 102, the PD 104, andthe transistor 105 forms one pixel.

A microlens 101 is provided, corresponding to each pixel, on theincident light incidence side of the color filter 102. The microlens 101condenses incident light toward the corresponding PD 104.

The interconnection layer 108 has interconnections 107 that transmit apixel signal from the PD layer 106 to the signal processing chip 111.The interconnection 107 may be a multilayer, and may be provided with apassive element and an active element.

A plurality of the bumps 109 is disposed on a surface of theinterconnection layer 108. The plurality of bumps 109 are aligned with aplurality of the bumps 109 that are provided on the opposing surface ofthe signal processing chip 111, and, for example, the imaging chip 113and the signal processing chip 111 are pressed against each other;thereby, the aligned bumps 109 are bonded and electrically connectedwith each other.

Similarly, a plurality of the bumps 109 are disposed on the mutuallyopposing surfaces of the signal processing chip 111 and the memory chip112. These bumps 109 are aligned with each other, and, for example, thesignal processing chip 111 and the memory chip 112 are pressed againsteach other; thereby, the aligned bumps 109 are bonded and electricallyconnected with each other.

Note that bonding between the bumps 109 is not limited to Cu bumpbonding by solid phase diffusion, but microbump joining by soldermelting may be adopted. Also, approximately one bump 109 may beprovided, for example, for each unit group described below. Accordingly,the size of the bumps 109 may be larger than the pitch of the PDs 104.Also, in a peripheral area other than a pixel area where pixels arearrayed, a bump that is larger than the bumps 109 corresponding to thepixel area may also be provided.

The signal processing chip 111 has a TSV (through-silicon via) 110 thatconnects circuits that are provided on a frontside and a backside,respectively. The TSV 110 is preferably provided in the peripheral area.Also, the TSV 110 may be provided also in the peripheral area of theimaging chip 113, and the memory chip 112.

FIG. 2 is a diagram for explaining a pixel array and a unit group 131 ofthe imaging chip 113. In particular, the figure shows a state of theimaging chip 113 as observed from the backside. A matrix of twentymillion pixels or more is arrayed in the pixel area. In the presentembodiment, adjacent four pixels (four pixels, 16 pixels, form one unitgroup 131. Grid lines in the figure show the concept that adjacentpixels are grouped to form the unit group 131. The number of pixels thatform the unit group 131 is not limited thereto, but may be approximately1000, for example thirty two pixels (sixty four pixels, or more or less.

As illustrated in the partially enlarged view of the pixel area, theunit group 131 includes, within its upper left, upper right, lower left,and lower right portions, four so-called Bayer arrays each includingfour pixels including green pixels Gb, Gr, a blue pixel B, and a redpixel R. The green pixels have green filters as the color filters 102,and receive light in the green wavelength band of incident light.Similarly, the blue pixel has a blue filter as the color filter 102, andreceives light in the blue wavelength band, and the red pixel has a redfilter as the color filter 102, and receives light in the red wavelengthband.

In the present embodiment, at least one unit group among a plurality ofthe unit groups 131 is selected, and pixels included in each unit groupare controlled according to control parameters that are different fromthose for other unit groups. Examples of the control parameters includea frame rate, a thinning rate, the number of added rows or the number ofadded columns whose pixel signals are added, a period or the number oftimes of accumulating electrical charges, the number of bits fordigitization, and the like. Furthermore, the control parameters may beparameters in image processing performed after acquiring image signalsfrom a pixel.

FIG. 3 is a schematic that corresponds to the unit group 131 of theimaging chip 113. In the figure, a rectangle that is indicated withdotted lines representatively represents a circuit that corresponds toone pixel. Note that at least a part of each transistor explained belowcorresponds to the transistor 105 in FIG. 1.

As described above, the unit group 131 is formed with 16 pixels. The 16PDs 104 that correspond to respective pixels are connected withrespective transfer transistors 302, and the gate of each transfertransistor 302 is connected with a TX interconnection 307 to whichtransfer pulses are supplied. In the present embodiment, the TXinterconnection 307 is connected in common to the 16 transfertransistors 302.

The drain of each transfer transistor 302 is connected with the sourceof each corresponding reset transistor 303, and also a so-calledfloating diffusion FD between the drain of the transfer transistor 302and the source of the reset transistor 303 is connected with the gate ofan amplifying transistor 304. The drain of the reset transistor 303 isconnected with a Vdd interconnection 310 to which power supply voltageis supplied, and its gate is connected with a reset interconnection 306to which reset pulses are supplied. In the present embodiment, the resetinterconnection 306 is connected in common to the 16 reset transistors303.

The drain of each amplifying transistor 304 is connected with the Vddinterconnection 310 to which power supply voltage is supplied. Also, thesource of each amplifying transistor 304 is connected with the drain ofeach corresponding selecting transistor 305. The gate of each selectingtransistor is connected with a decoder interconnection 308 to whichselection pulses are supplied. In the present embodiment, the decoderinterconnection 308 is provided independently to each of the 16selecting transistors 305. Then, the source of each selecting transistor305 is connected with a common output interconnection 309. A loadcurrent source 311 supplies current to the output interconnection 309.That is, the output interconnection 309 for the selecting transistors305 is formed by a source follower. Note that the load current source311 may be provided on the imaging chip 113 side or on the signalprocessing chip 111 side.

Here, a flow from the start of electrical charge accumulation to pixeloutput after the end of the accumulation will be explained. When resetpulses are applied to the reset transistor 303 through the resetinterconnection 306, and simultaneously transfer pulses are applied tothe transfer transistor 302 through the TX interconnection 307,potential of the PD 104 and the floating diffusion FD is reset.

When the application of the transfer pulses is stopped, the PD 104converts received incident light into electrical charges, which are thenaccumulated. Thereafter, when transfer pulses are applied again in astate where reset pulses are not being applied, accumulated electricalcharges are transferred to the floating diffusion FD, and the potentialof the floating diffusion FD changes from reset potential to signalpotential after electrical charge accumulation. Then, when selectionpulses are applied to the selecting transistor 305 through the decoderinterconnection 308, variation in the signal potential of the floatingdiffusion FD is transmitted to the output interconnection 309 via theamplifying transistor 304 and the selecting transistor 305. Thereby,pixel signals corresponding to the reset potential and the signalpotential are output from the unit pixel to the output interconnection309.

As illustrated, in the present embodiment, the reset interconnection 306and the TX interconnection 307 are common to the 16 pixels that form theunit group 131. That is, the reset pulses and the transfer pulses are,respectively, applied simultaneously to all the 16 pixels. Accordingly,all the pixels that form the unit group 131 start electrical chargeaccumulation at the same timing, and end electrical charge accumulationat the same timing. Note that however pixel signals that correspond toaccumulated electrical charges are output selectively to the outputinterconnection 309 upon sequential application of selection pulses tothe respective selecting transistors 305. Also, the resetinterconnection 306, the TX interconnection 307, and the outputinterconnection 309 are provided separately for each unit group 131.

By configuring a circuit on the basis of the unit group 131 in thismanner, an electrical charge accumulation period can be controlled foreach unit group 131. In other words, adjacent unit groups 131 can becaused to output pixel signals for different electrical chargeaccumulation periods. Furthermore, by causing one unit group 131 torepeat electrical charge accumulation several times and output a pixelsignal at each time while another unit group 131 is caused to performelectrical charge accumulation once, these unit groups 131 can be causedto output respective frames for a motion image at different frame rates.

FIG. 4 is a block diagram that illustrates a functional configuration ofthe imaging element 100. An analog multiplexer 411 sequentially selectsthe 16 PDs 104 that form the unit group 131, and causes their respectivepixel signals to be output to the output interconnection 309 providedcorresponding to the unit group 131. The multiplexer 411 is formed inthe imaging chip 113 together with the PDs 104.

A pixel signal that is output via the multiplexer 411 is subjected tocorrelated double sampling (CDS) and analog/digital (A/D) conversion bya signal processing circuit 412 that is formed in the signal processingchip 111 and performs CDS and A/D conversion. The A/D converted pixelsignal is passed over to a de-multiplexer 413, and is stored in thepixel memory 414 that corresponds to the respective pixel. Each pixelmemory 414 has a capacity that allows storage of pixel signals thatcorrespond to the maximum number of times of integration describedbelow. The de-multiplexer 413 and the pixel memory 414 are formed in thememory chip 112.

An arithmetic circuit 415 processes the pixel signal stored in the pixelmemory 414, and passes it over to an image processing unit in asubsequent step. The arithmetic circuit 415 may be provided in thesignal processing chip 111 or the memory chip 112. Note that although,in the figure, connections for a single unit group 131 are illustrated,connections actually exist for each unit group 131, and operate inparallel. Note that however the arithmetic circuit 415 may not exist foreach unit group 131, and, for example, a single arithmetic circuit 415may sequentially perform processing by sequentially referring to valuesof the pixel memories 414 that correspond to the respective unit groups131.

As described above, the output interconnection 309 is providedcorresponding to each of the unit groups 131. Because the imagingelement 100 is formed by layering the imaging chip 113, the signalprocessing chip 111, and the memory chip 112, the output interconnection309 can be routed without increasing the size of each chip in the planedirection by using inter-chip electrical connections that use the bumps109 for the interconnection.

FIG. 5 is a block diagram illustrating a configuration of an imagingdevice according to the present embodiment. An imaging device 500includes an imaging lens 520 as an imaging optical system, and theimaging lens 520 guides a subject luminous flux that is incident alongan optical axis OA to the imaging element 100. The imaging lens 520 maybe a replaceable lens that can be attached/detached to and from theimaging device 500. The imaging device 500 includes, mainly, the imagingelement 100, a system control unit 501, a drive unit 502, a photometryunit 503, a work memory 504, a recording unit 505, and a display unit506.

The imaging lens 520 is configured with a plurality of optical lensgroups, and forms an image of a subject luminous flux from a scene nearits focal plane. Note that, in FIG. 5, the imaging lens 520 isrepresentatively shown with a single virtual lens that is placed nearthe pupil. The drive unit 502 is a control circuit that executeselectrical charge accumulation control such as timing control and areacontrol on the imaging element 100 according to instructions from thesystem control unit 501. In this sense, it can be said that the driveunit 502 serves functions of an imaging element control unit that causesthe imaging element 100 to execute electrical charge accumulation andoutput pixel signals.

The imaging element 100 passes pixel signals over to an image processingunit 511 of the system control unit 501. The image processing unit 511performs various types of image processing by using the work memory 504as a workspace, and generates image data. For example, when image datain a JPEG file format is generated, compression processes are executedafter color video signals are generated from signals obtained from Bayerarrays. The generated image data is recorded in the recording unit 505and converted into display signals, and is displayed on the display unit506 for a preset period of time.

The photometry unit 503 detects luminance distribution of a scene priorto an imaging sequence for generating image data. The photometry unit503 includes an AE sensor of approximately one million pixels, forexample. A computing unit 512 of the system control unit 501 calculatesluminance of respective areas within a scene, upon receiving an outputof the photometry unit 503. The computing unit 512 decides a shutterspeed, a diaphragm value, and an ISO speed according to the calculatedluminance distribution. The imaging element 100 may double as thephotometry unit 503. Note that the computing unit 512 executes varioustypes of computation for operating the imaging device 500.

The drive unit 502 may be partially or entirely mounted on the imagingchip 113, or partially or entirely mounted on the signal processing chip111. The system control unit 501 may be partially mounted on the imagingchip 113 or the signal processing chip 111.

FIG. 6 is a functional block diagram of the image processing unit. Theimage processing unit 511 has, in addition to the above-describedfunctions, a subject estimating unit 150, a group selecting unit 152, amotion image generating unit 154, and a motion image synthesizing unit156. Each of these functions is described below.

FIG. 7 is a flowchart that illustrates operations of an imaging deviceto generate and record a motion image. FIGS. 8 and 9 each illustrate oneexample of an image imaged by an imaging element. FIG. 10 illustrates arelationship between respective frame rates and output timing of imagesignals.

Operations in FIG. 7 start when a user instructs the imaging device 500to generate a motion image for example by pressing down a record button.First, the subject estimating unit 150 drives the drive unit 502 toacquire image data based on image signals from the imaging element 100,and estimate a main subject included in an image indicated by the imagedata (S100).

In this case, the drive unit 502 preferably causes image signals fromunit groups 131 included in an entire imaging area, for example all theunit groups 131, to be output. Also, the drive unit 502 may cause imagesignals from all the pixels included in each unit group 131 to beoutput, or causes image signals from pixels that are thinned at apredetermined thinning rate to be output. The subject estimating unit150 compares a plurality of images obtained from the imaging element 100in a time-series, and identifies a moving subject as a main subject.Note that another method may be used to estimate a main subject.

For example, when the subject estimating unit 150 acquires an image 170in FIG. 8 and an image 178 in FIG. 9 from the imaging element 100 astemporally sequential images, based on differences therebetween, thesubject estimating unit 150 identifies a child as a main subject 171.Note that grid lines in the image 170 and the image 178 indicateboundaries of the unit groups 131, but the number of the unit groups 131is merely an example, and is not limited to the number shown in thefigures.

The group selecting unit 152 selects at least one unit group 131 onwhich image light of the main subject 171 estimated by the subjectestimating unit 150 is incident (S102). For example, unit groups 131including at least a part of the main subject 171 are selected in theimage 170. Furthermore, considering that the main subject 171 moves inan imaging area, the group selecting unit 152 preferably selects unitgroups 131 that further surround the unit groups 131 including at leasta part of the main subject 171.

The group selecting unit 152 handles a set of these selected unit groups131 as an attention area 172. Furthermore, the group selecting unit 152handles, as a peripheral area 176, a set of unit groups 131 not includedin the attention area 172 in the entire imaging area. The groupselecting unit 152 identifies area information 174 that indicates arange of the attention area 172 in relation to the entire imaging area.

In the example illustrated in FIG. 8, the attention area 172 is arectangular area including total 28 unit groups 131 (seven in thehorizontal direction (four in the vertical direction). On the otherhand, the peripheral area 176 includes 98 unit groups 131 excluding theattention area 172 from total 126 unit groups 131 (21 in the horizontaldirection (six in the vertical direction) which constitute the imagingarea. Also, the position (9, 2) of the attention area 172 in the imagingarea that is counted from the left side and the upper side of the upperleft end unit group 131 in the figure is identified as the areainformation 174. Furthermore, the numbers in the horizontal and verticaldirections, 7 (4, of the attention area 172 are identified as sizeinformation.

The group selecting unit 152 transmits information for identifying theunit groups 131 included in the attention area 172, and information foridentifying the peripheral area 176 to the drive unit 502. In this case,information on frame rates to be applied to the attention area 172 andthe peripheral area 176, respectively, is transmitted together. Here,the frame rate to be applied to the attention area 172 is preferablyhigher than the frame rate to be applied to the peripheral area 176. Forexample, when the frame rate to be applied to the peripheral area 176 is60 fps, the frame rate to be applied to the attention area 172 is set to180 fps. Preferably, values of the frame rates are preset, and storedsuch that the group selecting unit 152 can refer to them, but may bechangeable with an operation of a user afterwards.

The drive unit 502 drives the imaging element 100 to perform imaging atthe respective frame rates (S104). That is, the drive unit 502 causesthe unit groups 131 included in the attention area 172 to executeelectrical charge accumulation and image signal output at a high framerate, and causes the unit groups 131 included in the peripheral area 176to execute electrical charge accumulation and image signal output at alow frame rate. In other words, the drive unit 502 obtains image signalsthat correspond to a plurality of frames that are contiguous in atime-series for the unit groups 131 included in the attention area 172while obtaining image signals that correspond to a single frame for theunit groups 131 included in the peripheral area 176.

For example, when the frame rate of the peripheral area 176 is set to 60fps and the frame rate of the attention area 172 is set to 180 fps, asillustrated in FIG. 10, the drive unit 502 obtains image signals ofthree frames A1, A2, A3 from the attention area 172 during time 1/60 sin which image signals of a single frame B1 from the peripheral area 176are obtained ( 1/60 s=3× 1/180 s). In this case, the drive unit 502obtains image signals at different frame rates by separately driving aset of the reset transistors 303, the transfer transistors 302, and theselecting transistors 305 of the unit groups 131 included in theperipheral area 176, and a set of the reset transistors 303, thetransfer transistors 302, and the selecting transistors 305 of the unitgroups 131 included in the attention area 172.

Note that FIG. 10 illustrates timing of outputting image signals, butdoes not illustrate length of an exposure period. The drive unit 502drives the above-described sets of the transistors for the peripheralarea 176 and for the attention area 172 such that the exposure periodpreviously calculated by the computing unit 512 can be attained.

In addition to this, the length of the exposure period may be changedaccording to frame rates. For example, in the example illustrated inFIG. 10, the exposure period of one frame of the peripheral area 176 maybe set to ⅓, which is substantially the same with that for the attentionarea 172. Also, image signals may be corrected by the ratio of the framerates after outputting the image signals. Also, the timing of outputtingimage signals may not be synchronous as in FIG. 10, but may beasynchronous between the peripheral area 176 and the attention area 172.

The image processing unit 511 sequentially stores, on a frame-by-framebasis, image signals from the attention area 172 in a predeterminedstorage area of the work memory 504 (S106). Similarly, the imageprocessing unit 511 sequentially stores, on a frame-by-frame basis,image signals from the peripheral area 176 in a predetermined storagearea of the work memory 504 (the same step).

The motion image generating unit 154 reads out the image signals of theattention area 172 stored in the work memory 504 (S108), and generatesdata of the attention area motion image which includes a plurality offrames of the attention area 172 (S110). Similarly, the motion imagegenerating unit 154 reads out the image signals of the peripheral area176 stored in the work memory 504, and generates data of the peripheralarea motion image which includes a plurality of frames of the peripheralarea 176 (the same step). Here, the attention area motion image and theperipheral area motion image may each be generated in general-purposeformats such as MPEG and be able to be reproduced separately, or mayeach be generated in dedicated formats that do not allow reproductionwithout going through synthesis processing described below.

FIG. 11 schematically illustrates an attention area motion image and aperipheral area motion image generated by the motion image generatingunit. The motion image generating unit 154 generates the attention areamotion image at a frame rate that corresponds to a frame rate at whichthe drive unit 502 drove the attention area 172. In the exampleillustrated in FIG. 11, the attention area motion image is generated atthe frame rate 1/180 fps which is the same with the frame rate 1/180 fpsat which the drive unit 502 drove the attention area 172.

Similarly, the motion image generating unit 154 generates the peripheralarea motion image at a frame rate that corresponds to a frame rate atwhich the drive unit 502 drove the peripheral area 176. In the exampleillustrated in FIG. 11, the peripheral area motion image is generated atthe frame rate 1/60 fps which is the same with the frame rate 1/60 fpsat which the drive unit 502 drove the peripheral area 176. Note thateffective values do not exist in an area of the peripheral area motionimage that corresponds to the attention area 172, and the area isindicated with diagonal lines in the figure.

Furthermore, the motion image generating unit 154 adds headerinformation to the attention area motion image and the peripheral areamotion image, and records the data in the recording unit 505 (S112). Theheader information includes the area information that indicates theposition of the attention area 172 in relation to the entire imagingarea, the size information that indicates the size of the attention area172, and timing information that indicates a relationship between outputtiming of image signals of the attention area 172 and output timing ofimage signals of the peripheral area 176.

The system control unit 501 determines whether to perform imaging for anext unit time (S114). Whether to perform imaging of a next unit time isdetermined based on whether, at the time point, a user is pressing downa motion image record button. When imaging is to be performed for a nextunit time (S114: Yes), the flow returns to the above-described StepS102, and when imaging is not to be performed for the next unit time(S114: No), the operation ends.

Here, the “unit time” is preset in the system control unit 501, andlasts for several seconds. The storage capacity used for storage at StepS106 is determined based on this unit time, the frame rate and number ofunit groups of the attention area 172, and the frame rate and number ofunit groups of the peripheral area 176. Based also on these pieces ofinformation, an area of the storage capacity that stores data of theattention area 172 and an area of the storage capacity that stores dataof the peripheral area 176 are determined.

In this manner, image signals can be obtained at a high frame rate fromthe attention area 172 including the main subject 171, and also a dataamount can be reduced by keeping the frame rate for the peripheral area176 low. Accordingly, as compared with high speed readout from all thepixels, loads of driving and image processing can be reduced, and powerconsumption and heat generation can be suppressed.

Note that when a next unit time starts in the example illustrated inFIG. 7, unit groups 131 are selected again at Step S102, and the areainformation and the size information are updated. Thereby, the attentionarea 172 can be updated successively by tracking the main subject 171.In the example illustrated in FIG. 11, in a first frame A7 of the unittime in the attention area motion image, an attention area 182 includingunit groups 131 that are different from those of a last frame A6 in theprevious unit time are selected, and in accordance with this, areainformation 184 and a peripheral area 186 are updated.

FIG. 12 illustrates one example of the header information added by themotion image generating unit. The header information in FIG. 12 includesattention area motion image IDs that identify attention area motionimages, frame rates of the attention area motion images, peripheral areamotion image IDs that identify peripheral area motion imagescorresponding to the attention area motion images, frame rates of theperipheral area motion images, timing information, area information, andsize information. These pieces of the header information may be added asthe header information to either one or both of the attention areamotion image and the peripheral area motion image.

FIG. 13 is a flowchart that illustrates operations of an imaging deviceto reproduce and display a motion image. The operations start when auser specifies any of attention area motion images displayed asthumbnails on the display unit 506, and presses down a reproductionbutton.

The motion image synthesizing unit 156 reads out, from the recordingunit 505, data of an attention area motion image specified by the user(S150). The motion image synthesizing unit 156 reads out, from therecording unit 505, data of a peripheral area motion image correspondingto the attention area motion image (S152).

In this case, the motion image synthesizing unit 156 identifies theperipheral area motion image based on a peripheral area motion image IDindicated in the header information of the attention area motion imageread out at Step S150. Instead of this, a peripheral area image thatincludes, as the header information, timing information which is thesame with the timing information indicated in the header information ofthe attention area motion image may be searched for and identified.

Note that the header information is included in the attention areamotion image in the above-described example. On the other hand, when theheader information is not included in the attention area motion image,but in the peripheral area motion image, the user may be, previously atStep S150, caused to specify the peripheral area motion image which isto be read out, and the attention area motion image is specified andread out from the header information at Step S152.

The motion image synthesizing unit 156 synthesizes a frame of theattention area motion image and a frame of the peripheral area motionimage into a frame of a displayed motion image (S154). In this casefirst, the first frame A1 of the attention area motion image is fittedat a position indicated by the area information 174 in the first frameB1 of the peripheral area motion image to form a synthesized first frameC1 of the displayed motion image. As illustrated in FIG. 11, the motionimage synthesizing unit 156 causes the first frame C1 of the displayedmotion image to be displayed on the display unit 506 (S156).

The motion image synthesizing unit 156 determines whether there is anext frame of the attention area motion image before a next frame B2 ofthe peripheral area motion image (S158). When there is a next frame ofthe attention area motion image (S158: Yes), the motion imagesynthesizing unit 156 updates the attention area 172 by using the nextframes A2, A3, and keeps the peripheral area 176 at the previous frameB1 (S162) to form next synthesized frames C2, C3 of the displayed motionimage (S162), and display them sequentially (S156).

On the other hand, when there is not a next frame of the attention areamotion image before the next frame B2 of the peripheral area motionimage at Step S158 (S158), the motion image synthesizing unit 156updates the attention area 172 by using a next frame A4 and updates alsothe peripheral area 176 by using the next frame B2 (S164) to form a nextsynthesized frame C4 of the displayed motion image (S162), and displayit (S156).

As long as there is a next frame of the peripheral area 176 in theperipheral area motion image (S160: Yes), Steps S154 to S160 arerepeated. When there is not a next frame of the peripheral area 176 inthe peripheral area motion image (S160: No), the motion imagesynthesizing unit 156 makes a search to determine whether, at a unittime next to the unit time of the set of the attention area motion imageand the peripheral area motion image, there is a set of an attentionarea motion image and a peripheral area motion image (S166). Forexample, the motion image synthesizing unit 156 makes a search in thesame folder of the recording unit 505 to determine whether there isanother attention area motion image whose header information includestiming information indicating timing that immediately follows timingindicated by timing information of the previous attention area motionimage.

As long as there is a set of an attention area motion image and aperipheral area motion image in a next unit time (S166: Yes), Steps S150to S166 are repeated. When there is not a set of an attention areamotion image and a peripheral area motion image in a next unit time(S166: No), the operation ends.

In this manner, a smooth motion image can be displayed about theattention area 172 in which the main subject 171 is included whilereducing the overall data amount. Note that although at Step S162, theattention area 172 is updated directly by using the next frames to formthe synthesized frames of the displayed image, the method of synthesisis not limited thereto. As another example, the boundary line of themain subject 171 in the attention area 172 may be identified by imageprocessing, the main subject 171 surrounded by the boundary line may beupdated with a next frame, and the outside of the boundary line of themain subject 171 may be kept at the previous frame even if it is withinthe attention area 172, to form a synthesized frame with the peripheralarea 176. That is, the frame rate of the outside of the boundary line inthe attention area 172 may be lowered to the frame rate of theperipheral area 176. Thereby, it is possible to prevent boundaries ofsmoothness in the displayed motion image from looking unnatural. Also,the frame rates of reproduction need not be the same with the framerates at the time of imaging (180 fps for the attention area, and 60 fpsfor the peripheral area), but the frame rates may be for example 60 fpsand 20 fps for the attention area and the peripheral area, respectively.In such a case, the reproduction is slow-motion reproduction.

FIG. 14 is a flowchart that illustrates another example of operations ofthe imaging device to generate and record a motion image. Operations ofFIG. 14 that are the same with those of FIG. 7 are given the samereference numbers, and explanation thereof is omitted.

In the operations of FIG. 14, in addition to or instead of the framerates in FIG. 7, thinning rates are made different between the attentionarea 172 and the peripheral area 176. More specifically, at Step S120,the drive unit 502 causes the unit groups 131 included in the attentionarea 172 to execute electrical charge accumulation and image signaloutput of pixels that are thinned at a low thinning rate, and causesunit groups 131 included in the peripheral area 176 to executeelectrical charge accumulation and image signal output of pixels thatare thinned at a high thinning rate. For example, pixels in the unitgroups 131 included in the attention area 172 that are thinned at thethinning rate of 0, that is, all the pixels are read out, and pixels inthe unit groups 131 included in the peripheral area 176 that are thinnedat the thinning rate of 0.5, that is, a half of the pixels are read out.

In this case, the drive unit 502 obtains image signals at differentthinning rates by separately driving a set of the reset transistors 303,the transfer transistors 302, and the selecting transistors 305 of theunit groups 131 included in the peripheral area 176, and a set of thereset transistors 303, the transfer transistors 302, and the selectingtransistors 305 of the unit groups 131 included in the attention area172.

At Step S110, the motion image generating unit 154 generates anattention area motion image that corresponds to the attention area 172based on image signals of the attention area 172 output at a lowthinning rate. The motion image generating unit 154 similarly generatesa peripheral area motion image that corresponds to the peripheral area176 based on the image signals of the peripheral area 176 output at ahigh thinning rate. Also at Step S112, the motion image generating unit154 records the attention area motion image and the peripheral areamotion image, with information on the respective thinning rates beingadded thereto, in the recording unit 505.

FIG. 15 illustrates an example of pixels 188 to be read out at thethinning rate of 0.5 in one unit group. In the example illustrated inFIG. 15, when a unit group 132 in the peripheral area 176 is a Bayerarray, the pixels 188 to be read out and pixels not to be read out areset for every other Bayer array, that is, every two pixels alternatelyin the vertical direction. Thereby, thinned readout can be performedwithout losing a color balance.

FIG. 16 is a flowchart that illustrates operations, corresponding toFIG. 13, of the imaging device to reproduce and display a motion image.Operations of FIG. 16 that are the same with those of FIG. 13 are giventhe same reference numbers, and explanation thereof is omitted.

At Step S170 in FIG. 16, the motion image synthesizing unit 156complements pixels of a frame of the peripheral area motion image tomatch its resolution with the resolution of a frame of the attentionarea motion image, and thereafter fits the frame of the attention areamotion image to the frame of the peripheral area motion image; thereby,a synthesized frame of the displayed image is formed. Thereby, imagesignals can be obtained at a high resolution from the attention area 172including the main subject 171, and also the data amount can be reducedby keeping the resolution of the peripheral area 176 low. Accordingly,as compared with high speed readout from all the pixels, loads ofdriving and image processing can be reduced, and power consumption andheat generation can be suppressed.

Note that although the attention area 172 is a rectangle in the examplesillustrated in FIGS. 1 to 16, the shape of the attention area 172 is notlimited thereto. The attention area 172 may be a convex or concavepolygon, or may have a doughnut shape with the peripheral area 176positioned inside thereof or another shape as long as the attention area172 conforms to the boundary line of the unit group 131. Also, aplurality of the attention areas 172 that are spaced apart from eachother may be set. In such a case, mutually different frame rates may beset for the attention areas 172.

Also, frame rates of the attention area 172 and the peripheral area 176may be variable. For example, the moving amount of the main subject 171may be detected with the elapse of a unit time, and a higher frame ratemay be set for the attention area 172 if the moving amount of the mainsubject 171 is larger. Also, selection of unit groups 131 that should beincluded in the attention area 172 may be updated at any time during theunit time, by tracking the main subject 171.

Although motion image generation in FIGS. 7 and 14 starts when a userpresses down a record button, and motion image reproduction in FIGS. 13and 16 starts when a user presses down a reproduction button, thestarting time points are not limited thereto. As another example,triggered by a single button operation by a user, an operation of motionimage generation and an operation of motion image reproduction may becontinuously executed, and a through-image display (or also called alive view display) may be performed on the display unit 506. In thiscase, a display for causing the user to recognize the attention area 172may be superimposed. For example, a frame may be displayed over theboundary of the attention area 172 on the display unit 506, or theluminance of the peripheral area 176 may be lowered or the luminance ofthe attention area 172 may be raised.

In the operations in FIG. 14, thinning rates are made different betweenthe attention area 172 and the peripheral area 176. Instead of makingthe thinning rates different, the numbers of adjacent rows of pixelswhose pixel signals are added may be made different. For example, in theattention area 172, the number of rows is one, which means that pixelsignals are output without addition among adjacent rows, and in theperipheral area 176, the number of rows is larger than that for theattention area 172, that is, for example two, which means that pixelsignals of pixels of two adjacent rows that are in the same columns areoutput. Thereby, similar to FIG. 14, the overall signal amount can bereduced while keeping the resolution of the attention area 172 higherthan that of the peripheral area 176. Also, instead of adding pixelsignals of adjacent rows, pixel signals of adjacent columns may beadded. In this case, the numbers of columns in adding pixel signals ofpixels of adjacent columns are made different between the attention area172 and the peripheral area 176. Furthermore, in the above-describedaddition, a process of calculating an average by dividing the sum valueby the number of added rows or columns may be included.

Note that the motion image synthesizing unit 156 may be provided in anexternal display apparatus, for example a PC, instead of being providedin the image processing unit 511 of the imaging device 500. Also, theabove-described embodiment may be applied not only to motion imagegeneration, but also to still image generation.

Also, although in the above-described embodiments, a plurality of theunit groups 131 is divided into two areas, the attention area 172 andthe peripheral area 176, the number of division is not limited thereto,and the unit groups 131 may be divided into three or more areas. In thiscase, unit groups 131 that correspond to the boundary between theattention area 172 and the peripheral area 176 may be handled as aboundary area, and the boundary area may be controlled by using anintermediate value between a value of a control parameter used for theattention area 172 and a value of a control parameter used for theperipheral area 176. Thereby, it is possible to prevent the boundarybetween the attention area 172 and the peripheral area 176 from lookingunnatural.

Accumulation periods and numbers of times of accumulation of electricalcharges, and the like may be made different between the attention area172 and the peripheral area 176. In this case, the attention area 172and the peripheral area 176 may be divided based on luminance, andfurthermore an intermediate area may be provided.

FIG. 17 is a diagram for explaining an example of a scene and areadivision. FIG. 17( a) illustrates a scene captured by a pixel area ofthe imaging chip 113. Specifically, the scene includes simultaneously ashadowed subject 601 and an intermediate subject 602 included in anindoor environment, and a highlighted subject 603 of an outdoorenvironment observed within a window frame 604. When imaging, with aconventional imaging element, such a scene in which the contrast betweena highlighted portion and a shadowed portion is high, blocked-up shadowsoccur at the shadowed portion if electrical charge accumulation isexecuted by using the highlighted portion as a reference, and blown-outhighlights occur at the highlighted portion if electrical chargeaccumulation is executed by using the shadowed portion as a reference.That is, it can be said that, for a high contrast scene, the photo diodedoes not have a sufficient dynamic range that is needed for imagesignals to be output by one-time electrical charge accumulation that isuniform for the highlighted portion and the shadowed portion. To copewith this, in the present embodiment, a scene is divided into partialareas such as a highlighted portion and a shadowed portion, andsubstantial expansion of a dynamic range is attempted by making thenumbers of times of electrical charge accumulation mutually differentbetween photo diodes that correspond to respective areas.

FIG. 17( b) illustrates area division of a pixel area in the imagingchip 113. The computing unit 512 analyzes the scene of FIG. 17( a)captured by the photometry unit 503 to divide the pixel area based onluminance. For example, the system control unit 501 causes thephotometry unit 503 to execute scene acquisition multiple times whilechanging exposure periods, and the computing unit 512 decides divisionlines of the pixel area by referring to changes in distribution ofblown-out highlight areas and blocked-up shadowed areas. In the exampleof FIG. 17( b), the computing unit 512 performs division into threeareas, a shadowed area 611, an intermediate area 612, and a highlightedarea 613.

The division line is defined along boundaries of unit groups 131. Thatis, each divided area includes an integer number of groups. Then, pixelsof each group included in the same area perform electrical chargeaccumulation and pixel signal output the same number of times in aperiod that corresponds to a shutter speed decided by the computing unit512. If pixels belong to different areas, electrical charge accumulationand pixel signal output are performed different numbers of times.

FIG. 18 is a diagram for explaining electrical charge accumulationcontrol for the respective areas divided in the example in FIG. 17. Uponreceiving an imaging stand-by instruction from a user, the computingunit 512 decides a shutter speed T₀ based on an output from thephotometry unit 503. Furthermore, the computing unit 512 performsdivision into the shadowed area 611, the intermediate area 612, and thehighlighted area 613 in a manner as above-described, and decides thenumbers of times of electrical charge accumulation based on respectivepieces of luminance information. The numbers of times of electricalcharge accumulation are decided such that pixels are not saturated byone-time electrical charge accumulation. For example, the numbers oftimes of electrical charge accumulation are decided such that 80 to 90%of accumulatable electrical charges is accumulated in a one-timeelectrical charge accumulation operation.

Here, electrical charge accumulation is performed once for the shadowedarea 611. That is, the decided shutter speed T₀ and the electricalcharge accumulation period are caused to match. Also, electrical chargeaccumulation is performed twice for the intermediate area 612. That is,a one-time electrical charge accumulation period is set to T₀/2, andelectrical charge accumulation is repeated twice during the shutterspeed T₀. Also, electrical charge accumulation is performed four timesfor the highlighted area 613. That is, a one-time electrical chargeaccumulation period is set to T₀/4, and electrical charge accumulationis repeated four times during the shutter speed T₀.

Upon receiving an imaging instruction from a user at a clock time t=0,the drive unit 502 applies reset pulses and transfer pulses to pixels ingroups belonging to the respective areas. This application triggers astart of electrical charge accumulation of all the pixels.

At a clock time t=T₀/4, the drive unit 502 applies transfer pulses topixels in groups belonging to the highlighted area 613. Then, the driveunit 502 sequentially applies selection pulses to pixels in each groupto cause their respective pixel signals to be output to the outputinterconnection 309. After pixel signals of all the pixels in the groupsare output, the drive unit 502 applies reset pulses and transfer pulsesagain to pixels in groups belonging to the highlighted area 613 to causesecond electrical charge accumulation to be started.

Note that because selective output of pixel signals takes time, a timelag occurs between the end of first electrical charge accumulation andthe start of second electrical charge accumulation. When this time lagis substantially negligible, a one-time electrical charge accumulationperiod may be calculated by dividing the shutter speed T₀ by the numbersof times of electrical charge accumulation as described above. On theother hand, if not negligible, the shutter speed T₀ may be adjusted byconsidering the time, or the a one-time electrical charge accumulationperiod may be made shorter than the time obtained by dividing theshutter speed T₀ by the numbers of times of electrical chargeaccumulation.

At a clock time t=T₀/2, the drive unit 502 applies transfer pulses topixels in groups belonging to the intermediate area 612 and thehighlighted area 613. Then, the drive unit 502 sequentially appliesselection pulses to pixels in each group to cause their respective pixelsignals to be output to the output interconnection 309. After pixelsignals of all the pixels in the groups are output, the drive unit 502applies reset pulses and transfer pulses again to pixels in groupsbelonging to the intermediate area 612 and the highlighted area 613 tocause second electrical charge accumulation to be started for theintermediate area 612 and cause third electrical charge accumulation tobe started for the highlighted area 613.

At a clock time t=3T₀/4, the drive unit 502 applies transfer pulses topixels in groups belonging to the highlighted area 613. Then, the driveunit 502 sequentially applies selection pulses to pixels in each groupto cause their respective pixel signals to be output to the outputinterconnection 309. After pixel signals of all the pixels in the groupsare output, the drive unit 502 applies reset pulses and transfer pulsesagain to pixels in groups belonging to the highlighted area 613 to causefourth electrical charge accumulation to be started.

At the clock time t=T₀, the drive unit 502 applies transfer pulses topixels of all the areas. Then, the drive unit 502 sequentially appliesselection pulses to pixels in each group to cause their respective pixelsignals to be output to the output interconnection 309. According to theabove-described control, pixel signals that correspond to once arestored in each pixel memory 414 that corresponds to the shadowed area611, pixel signals that correspond to twice are stored in each pixelmemory 414 that corresponds to the intermediate area 612, and pixelsignals that correspond to four times are stored in each pixel memory414 that corresponds to the highlighted area 613.

These pixel signals are sequentially transferred to the image processingunit 511. The image processing unit 511 generates image data with a highdynamic range based on the pixel signals. Specific processing isdescribed below.

FIG. 19 is a table that indicates a relationship between the number oftimes of integration and the dynamic range. Pixel signals thatcorrespond to multiple times of repeatedly executed electrical chargeaccumulation are subjected to an integration process by the imageprocessing unit 511 to form a part of image data with a high dynamicrange.

When compared with, as a reference, a dynamic range of an area whosenumber of times of integration is once, that is, for which electricalcharge accumulation is performed once, a dynamic range of an area whosenumber of times of integration is twice, that is, whose output signal isintegrated by performing electrical charge accumulation twice isexpanded by one step. Similarly, when the number of times of integrationis four times, the dynamic range is expanded by two steps, and when thenumber of times of integration is 128, the dynamic range is expanded byseven steps. That is, in order to attempt to obtain n-steps of dynamicrange expansion, output signals may be integrated 2^(n) times.

Here, in order for the image processing unit 511 to identify how manytimes electrical charge accumulation has been performed for whichdivided area, a 3-bit exponent indicating the number of times ofintegration is added to an image signal. As illustrated, exponents areallocated sequentially, 000 to the number of times of integration once,001 to twice, . . . , 111 to 128 times.

The image processing unit 511 refers to an exponent of each pixel signalreceived from the arithmetic circuit 415 and when a result of thereference shows that the number of times of integration is two or more,executes an integration process of the pixel signal. For example, whenthe number of times of integration is two (one step), upper 11 bits oftwo 12-bit pixel signals corresponding to electrical charge accumulationare added together to generate a single 12-bit pixel signal. Similarly,when the number of times of integration is 128 (seven steps), upper 5bits of 128 12-bit pixel signals corresponding to electrical chargeaccumulation are added together to generate a single 12-bit pixelsignal. That is, upper bits, the number of which is obtained bysubtracting, from 12, the number of steps corresponding to the number oftimes of integration, are added together to generate a single 12-bitpixel signal. Note that lower bits that are not to be added areeliminated.

By performing processing in this manner, the luminance range thatprovides a gradation can be shifted to the high luminance side inaccordance with the number of times of integration. That is, 12 bits areallocated to a limited range on the high luminance side. Accordingly, agradation can be provided to an image area that conventionally includedblown-out highlights.

Note that however that, because 12 bits are allocated to differentluminance ranges of other divided areas, image data cannot be generatedby synthesis of simply connecting the areas. To cope with this, theimage processing unit 511 performs a re-quantization process by using,as a reference, a highest luminance pixel and a lowest luminance pixelin order to make all the areas 12-bit image data while preservingobtained gradations as much as possible. Specifically, quantization isexecuted by performing gamma conversion so that the smoother gradationscan be preserved. By performing processing in this manner, image datawith a high dynamic range can be obtained.

Note that the description of the number of times of integration is notlimited to a 3-bit exponent being added to a pixel signal asabove-described, but the number of times of integration may be describedas accompanying information other than the pixel signal. Also, theexponent may be omitted from a pixel signal, and instead the number oftimes of integration may be acquired at the time of an adding process bycounting the number of pixel signals stored in the pixel memory 414.

Also, although in the above-described image processing, are-quantization process to make all the areas 12-bit image data isexecuted, the number of output bits may be increased from the bit numberof a pixel signal, in accordance with an upper limit number of times ofintegration. For example, if the upper limit number of times ofintegration is defined as 16 (four steps), all the areas may be made,for a 12-bit pixel signal, 16-bit image data. By performing processingin this manner, image data can be generated without cancellation ofdigits.

Next, a series of imaging operation processes is explained. FIG. 20 is aflow diagram showing processing of imaging operations. The flow startswhen a power supply of the imaging device 500 is turned on.

At Step S201, the system control unit 501 waits for a switch SW1 to bepressed down, which is an imaging stand-by instruction. When pressingdown of the switch SW1 is sensed, the flow proceeds to Step S202.

At Step S202, the system control unit 501 executes photometryprocessing. Specifically, upon obtaining an output of the photometryunit 503, the computing unit 512 calculates luminance distribution of ascene. Then, the flow proceeds to Step S203, and as described above, ashutter speed, area division, the number of times of integration, andthe like are decided.

Upon completion of the imaging stand-by operation, the flow proceeds toStep S204, and waits for a switch SW2 to be pressed down, which is animaging instruction. At this time, when the elapsed time exceeds apredetermined time Tw (YES at Step S205), the flow returns to Step S201.When pressing down of the switch SW2 is sensed before the elapsed timeexceeds the time Tw (NO at Step S205), the flow proceeds to Step S206.

At Step S206, the drive unit 502 that has received an instruction of thesystem control unit 501 executes an electrical charge accumulationprocess and a signal readout process that are explained by using FIG.18. Then, upon completion of entire signal readout, the flow proceeds toStep S207, the image processing explained by using FIG. 19 is executed,and a recording process of recording generated image data in therecording unit is executed.

Upon completion of the recording process, the flow proceeds to StepS208, and it is determined whether the power supply of the imagingdevice 500 has been turned off. When the power supply has not beenturned off, the flow returns to Step S201, and when the power supply hasbeen turned off, the series of imaging operation processes ends.

FIG. 21 is a block diagram that illustrates a specific configuration ofthe signal processing chip 111 as one example. Although one example inwhich the de-multiplexer 413 and the pixel memory 414 are formed in thememory chip 112 is explained by using FIG. 4 above, an example in whichthe de-multiplexer 413 and the pixel memory 414 are formed in the signalprocessing chip 111 is explained here.

The signal processing chip 111 serves functions of the drive unit 502.The signal processing chip 111 includes a sensor control unit 441, ablock control unit 442, a synchronization control unit 443, and a signalcontrol unit 444 that serve divided control functions, and a drivecontrol unit 420 that performs overall control on the respective controlunits. The drive control unit 420 converts instructions from the systemcontrol unit 501 into control signals that can be executed by therespective control units, and passes them over to the respective controlunits.

The sensor control unit 441 performs transmission control on controlpulses that are to be transmitted to the imaging chip 113 and relate toelectrical charge accumulation and electrical charge readout of eachpixel. Specifically, the sensor control unit 441 controls the start andend of electrical charge accumulation by transmitting reset pulses andtransfer pulses to target pixels, and causes pixel signals to be outputto the output interconnection 309 by transmitting selection pulses toreadout pixels.

The block control unit 442 executes transmission of specifying pulsesthat are to be transmitted to the imaging chip 113 and specify a unitgroup 131 to be controlled. As explained by using FIG. 17, etc., dividedareas may include a plurality of mutually adjacent unit groups 131. Unitgroups 131 belonging to the same area form a single block. Pixels thatare included in the same block start electrical charge accumulation atthe same timing, and end the electrical charge accumulation at the sametiming. To cope with this, the block control unit 442 plays a role offorming blocks of unit groups 131 by transmitting specifying pulses tounit groups 131 to be targets based on designation by the drive controlunit 420. Transfer pulses and reset pulses that each pixel receives viathe TX interconnection 307 and the reset interconnection 306 are logicalAND of each pulse transmitted by the sensor control unit 441 andspecifying pulses transmitted by the block control unit 442. In thismanner, by controlling each area as a mutually independent block, theelectrical charge accumulation control explained by using FIG. 18 can berealized. The block-formation designation by the drive control unit isdescribed in detail below.

The synchronization control unit 443 transmits a synchronization signalto the imaging chip 113. Each pulse becomes active in the imaging chip113 in synchronization with the synchronization signal. For example, byadjusting the synchronization signal, random control, thinning control,and the like only on particular pixels among pixels belonging to thesame unit group 131 can be realized.

The signal control unit 444 mainly performs timing control on an A/Dconverter 412 b. Pixel signals output via the output interconnection 309are input to the A/D converter 412 b through a CDS circuit 412 a and themultiplexer 411. The A/D converter 412 b is controlled by the signalcontrol unit 444 to convert the input pixel signals into digitalsignals. The pixel signals converted into the digital signals are passedover to the de-multiplexer 413, and are stored as a pixel value ofdigital data in the pixel memory 414 corresponding to each pixel.

The signal processing chip 111 has a timing memory 430, as anaccumulation control memory, that stores block division informationabout which unit groups 131 are to be combined to form a block, andinformation on the number of times of accumulation about how many timeseach block formed repeats electrical charge accumulation. The timingmemory 430 is configured for example with a flash RAM.

As described above, which unit groups are to be combined to form a blockis decided by the system control unit 501 based on a detection result ofluminance distribution detection of a scene that is executed prior to aseries of imaging sequence. The decided blocks are divided for exampleinto a first block, a second block, . . . , and defined by which unitgroups 131 are included therein. The drive control unit 420 receives theblock division information from the system control unit 501, and storesit in the timing memory 430.

Also, the system control unit 501 decides how many times each blockrepeats electrical charge accumulation based on a detection result ofluminance distribution. The drive control unit 420 receives theinformation on the number of times of accumulation from the systemcontrol unit 501, and stores it in the timing memory 430 by pairing theinformation on the number of times of accumulation with thecorresponding block division information. By storing the block divisioninformation and the information on the number of times of accumulationin the timing memory 430 in this manner, the drive control unit 420 mayexecute a series of electrical charge accumulation control independentlyby successively referring to the timing memory 430. That is, whencontrolling acquisition of a single image, once the drive control unit420 receives a signal of an imaging instruction from the system controlunit 501, the drive control unit 420 thereafter is able to completeaccumulation control without receiving an instruction about control oneach pixel from the system control unit 501 each time.

The drive control unit 420 receives, from the system control unit 501,block division information and information on the number of times ofaccumulation that are updated based on results of photometry (detectionresults of luminance distribution) executed in synchronization with animaging stand-by instruction, and as appropriate updates stored contentsof the timing memory 430. For example, the drive control unit 420updates the timing memory 430 in synchronization with an imagingstand-by instruction or an imaging instruction. With this configuration,faster electrical charge accumulation control is realized, and thesystem control unit 501 may execute other processing in parallel withelectrical charge accumulation control executed by the drive controlunit 420.

The drive control unit 420 which executes electrical charge accumulationcontrol on the imaging chip 113 further refers to the timing memory 430in execution of readout control. For example, the drive control unit 420refers to information on the number of times of accumulation of eachblock to store a pixel signal output from the de-multiplexer 413 in acorresponding address of the pixel memory 414.

The drive control unit 420 reads out a target pixel signal from thepixel memory 414 according to a delivery request from the system controlunit 501, and passes it over to the image processing unit 511. The pixelmemory 414 has a memory space that can store a pixel signalcorresponding to the maximum number of times of integration about eachpixel as described above, and stores, as pixel values, their respectivepixel signals corresponding to the number of times of accumulationexecuted. For example, because when electrical charge accumulation isrepeated four times in a block, pixels included in the block outputpixel signals that correspond to the four times, a memory space in thepixel memory 414 for each pixel stores four pixel values. When havingreceived, from the system control unit 501, a delivery request thatrequests a pixel signal of a particular pixel, the drive control unit420 specifies an address of the particular pixel on the pixel memory414, reads out all the stored pixel signals, and passes them over to theimage processing unit 511. For example when four pixel values arestored, all the four pixel values are sequentially passed over, and whenonly one pixel value is stored, the pixel value is passed over.

The drive control unit 420 can read out a pixel signal stored in thepixel memory 414, pass it to the arithmetic circuit 415, and cause thearithmetic circuit 415 to execute the above-described integrationprocess. The pixel signal having been subjected to the integrationprocess is stored in a target pixel address of the pixel memory 414. Thetarget pixel address may be provided adjacent to an address space beforethe integration process, or may be the same address so that a pixelsignal is written over the pixel signal before the integration process.Also, a dedicated space that collectively stores pixel values ofrespective pixels after the integration process may be provided. Whenhaving received, from the system control unit 501, a delivery requestthat requests a pixel signal of a particular pixel, the drive controlunit 420 can pass the pixel signal after the integration process over tothe image processing unit 511 depending on the form of the deliveryrequest. Of course, pixel signals before and after the integrationprocess may be passed over together.

A data transfer interface that transmits pixel signals according to adelivery request is provided to the pixel memory 414. The data transferinterface is connected with a data transfer line that connects with theimage processing unit 511. The data transfer line is configured forexample with a data bus among bus lines. In this case, a deliveryrequest from the system control unit 501 to the drive control unit 420is executed by addressing that utilizes an address bus.

Transmission of pixel signals by the data transfer interface is notlimited to an addressing system, but may adopt various systems. Forexample, at the time of data transfer, a double data rate system inwhich both rising and falling of a clock signal used for synchronizationof each circuit are utilized to perform processing may be adopted. Also,a burst transfer system of transferring data at once by partiallyomitting procedures such as addressing, and attempting speed up may beadopted. Also, a bus system of using lines that connect a control unit,a memory unit, and an input/output unit in parallel, and a serial systemof transferring data in series on a bit by bit basis may be adopted incombination.

With this configuration, because the image processing unit 511 canreceive only necessary pixel signals, the image processing unit 511 cancomplete image processing at high speed particularly when forming a lowresolution image. Also, because when the arithmetic circuit 415 iscaused to execute the integration process, the image processing unit 511does not have to execute the integration process, speeding up of theimage processing may be attempted by functional division and parallelprocessing.

In the above-described examples of FIGS. 17 to 21, by making the numbersof times of electrical charge accumulation and the like differentbetween the attention area 172 and the peripheral area 176, the numberof bits at the time when a pixel signal of the attention area 172 isdigitized is made larger than that for the peripheral area 176. Thenumbers of bits for digitization may be made different between theattention area 172 and the peripheral area 176 by another method. Forexample, an A/D circuit of the signal processing circuit 412 maydigitize the attention area 172 with a larger number of bits than theperipheral area 176 by the same one-time accumulation, according to aninstruction from the drive unit 502.

By using the signal processing chip 111 in FIG. 21, image processing maybe performed after acquiring a pixel signal by using control parametersthat are different between the attention area 172 and the peripheralarea 176. For example, although in FIGS. 7 to 10, a motion image isgenerated from images that are acquired at frame rates that aredifferent between the attention area 172 and the peripheral area 176,instead of this, an S/N ratio may be improved by performing imageprocessing of averaging images acquired at a high frame rate. In thiscase, the drive control unit 420 obtains pixel signals that correspondto multiple times, for example four times, from the attention area 142for example while obtaining pixel signals that corresponds to once fromthe peripheral area 176, and stores them in the pixel memory 414. Thearithmetic circuit 415 reads out a plurality of pixel signal obtained,from the pixel memory 414, for each pixel of the attention area 142, andaverages them for respective pixels. Thereby, random noises of eachpixel of the attention area 172 are reduced, and an S/N ratio of theattention area 172 can be improved.

Also, although in FIGS. 7 to 10, a motion image is generated from imagesthat are acquired at frame rates that are different between theattention area 172 and the peripheral area 176, frame rates may be madedifferent based on motion speeds of a subject. In this case, the subjectestimating unit 150 estimates speeds in the vertical and horizontaldirections based on changes in the position of a subject between frames.Also, the subject estimating unit 150 estimates speeds of a subject inthe front and rear directions based on changes in the size of thesubject between the frames. Based on the estimation, the group selectingunit 152 identifies unit groups 131 that receive light from a subjectmoving at low speed or a still subject, unit groups 131 that receivelight from a subject moving at an intermediate speed, and unit groups131 that receive light from a subject moving at high speed.

The drive unit 502 drives the imaging element 100 to perform imaging ofthe unit groups 131 that receive light from the subject moving at lowspeed or the still subject, the unit groups 131 that receive light fromthe subject moving at an intermediate speed, and the unit groups 131that receive light from the subject moving at high speed, at a low framerate, an intermediate frame rate, and a high frame rate, respectively.Examples of the frame rates are 60 fps, 120 fps, and 240 fps,respectively.

FIG. 22 is a sectional view of another backside illuminating typeimaging element 1100 according to the present embodiment. The imagingelement 1100 includes an imaging chip 1113 that outputs a pixel signalcorresponding to incident light, a signal processing chip 1111 thatprocesses the pixel signal, and a memory chip 1112 that stores the pixelsignal. These imaging chip 1113, signal processing chip 1111, and memorychip 1112 are layered, and are electrically connected with each othervia conductive bumps 1109, such as Cu.

Note that, as illustrated, incident light is incident mainly in the Zaxis positive direction that is indicated with an outlined arrow. In thepresent embodiment, the surface of the imaging chip 1113 on a side onwhich the incident light is incident is called a backside. Also, asindicated with coordinate axes, the leftward direction on the figurethat is orthogonal to the Z axis is referred to as the X axis positivedirection, and the front side direction in the figure that is orthogonalto the Z and X axes is referred to as the Y axis positive direction. Inseveral figures mentioned below, the coordinate axes are displayed suchthat the orientation of each figure can be known on the basis of thecoordinate axes in FIG. 22.

One example of the imaging chip 1113 is a backside illuminating type MOSimage sensor. A PD layer 1106 is disposed on a backside of aninterconnection layer 1108. The PD layer 1106 has a plurality of PDs(photo diodes) 1104 that are two-dimensionally disposed and accumulateelectrical charges according to incident light, and transistors 1105provided corresponding to the PDs 1104.

Color filters 1102 are provided on the incident light incidence side ofthe PD layer 1106 via a passivation film 1103. There is a plurality oftypes of the color filters 1102 that allow passage of mutually differentwavelength ranges, and the color filters 1102 are arrayed particularlycorresponding to the respective PDs 1104. The arrays of the colorfilters 1102 are described below. A set of the color filter 1102, the PD1104, and the transistor 1105 forms one pixel.

A microlens 1101 is provided, corresponding to each pixel, on theincident light incidence side of the color filter 1102. The microlens1101 condenses incident light toward the corresponding PD 1104.

The interconnection layer 1108 has interconnections 1107 that transmit apixel signal from the PD layer 1106 to the signal processing chip 1111.The interconnection 1107 may be a multilayer, and may be provided with apassive element and an active element.

A plurality of the bumps 1109 is disposed on a surface of theinterconnection layer 1108. The plurality of bumps 1109 are aligned witha plurality of the bumps 1109 that are provided on the opposing surfaceof the signal processing chip 1111, and, for example, the imaging chip1113 and the signal processing chip 1111 are pressed against each other;thereby, the aligned bumps 1109 are bonded and electrically connectedwith each other.

Similarly, a plurality of the bumps 1109 are disposed on the mutuallyopposing surfaces of the signal processing chip 1111 and the memory chip1112. These bumps 1109 are aligned with each other, and, for example,the signal processing chip 1111 and the memory chip 1112 are pressedagainst each other; thereby, the aligned bumps 1109 are bonded andelectrically connected with each other.

Note that bonding between the bumps 1109 is not limited to Cu bumpbonding by solid phase diffusion, but microbump joining by soldermelting may be adopted. Also, approximately one bump 1109 may beprovided, for example, for each unit group described below. Accordingly,the size of the bumps 1109 may be larger than the pitch of the PDs 1104.Also, in a peripheral area other than a pixel area where pixels arearrayed, a bump that is larger than the bumps 1109 corresponding to thepixel area may also be provided.

The signal processing chip 1111 has a TSV (through-silicon via) 1110that connects circuits that are provided on a frontside and a backside,respectively. The TSV 1110 is preferably provided in the peripheralarea. Also, the TSV 1110 may be provided also in the peripheral area ofthe imaging chip 1113, and the memory chip 1112.

FIG. 23 is a diagram for explaining a pixel array and a unit group 1131of the imaging chip 1113. In particular, the figure shows a state of theimaging chip 1113 as observed from the backside. A matrix of twentymillion pixels or more is arrayed in the pixel area. In the example ofFIG. 23, adjacent four pixels (four pixels, 16 pixels, form the unitgroup 1131. Grid lines in the figure show the concept that adjacentpixels are grouped to form the unit group 1131. The number of pixelsthat form the unit group 1131 is not limited thereto, but may beapproximately 1000, for example thirty two pixels (sixty four pixels, ormore or less.

As illustrated in the partially enlarged view of the pixel area, theunit group 1131 includes, within its upper left, upper right, lowerleft, and lower right portions, four so-called Bayer arrays eachincluding four pixels including green pixels Gb, Gr, a blue pixel B, anda red pixel R. The green pixels have green filters as the color filters1102, and receive light in the green wavelength band of incident light.Similarly, the blue pixel has a blue filter as the color filter 1102,and receives light in the blue wavelength band, and the red pixel has ared filter as the color filter 1102, and receives light in the redwavelength band.

In the present embodiment, an evaluation value is calculated for each ofa plurality of the unit groups 1131, and exposure or readout of pixelsincluded in the unit group is controlled by using control parametersbased on the evaluation value. Examples of the evaluation value includean average of pixel signals within a unit group 1131, a weighted averageof pixel signals within and outside a unit group 1131, contrast within aunit group 1131, a weighted average of contrast within and outside aunit group 1131, luminance within a unit group 1131, and a weightedaverage of luminance within and outside a unit group 1131. Examples ofthe control parameters include a frame rate, a thinning rate, the numberof added rows or the number of added columns whose pixel signals areadded, a period or the number of times of accumulating electricalcharges, the number of bits for digitization, and the like. Furthermore,the control parameters may be parameters in image processing performedafter acquiring image signals from a pixel.

FIG. 24 is a schematic that corresponds to the unit group 1131 of theimaging chip 1113. In the figure, a rectangle that is indicated withdotted lines representatively represents a circuit that corresponds toone pixel. Note that at least a part of each transistor explained belowcorresponds to the transistor 1105 in FIG. 22.

As described above, the unit group 1131 is formed with 16 pixels. The 16PDs 1104 that correspond to respective pixels are connected withrespective transfer transistors 1302, and the gate of each transfertransistor 1302 is connected with a TX interconnection 1307 to whichtransfer pulses are supplied. In the present embodiment, the TXinterconnection 1307 is connected in common to the 16 transfertransistors 1302.

The drain of each transfer transistor 1302 is connected with the sourceof each corresponding reset transistor 1303, and also a so-calledfloating diffusion FD between the drain of the transfer transistor 1302and the source of the reset transistor 1303 is connected with the gateof an amplifying transistor 1304. The drain of the reset transistor 1303is connected with a Vdd interconnection 1310 to which power supplyvoltage is supplied, and its gate is connected with a resetinterconnection 1306 to which reset pulses are supplied. In the presentembodiment, the reset interconnection 1306 is connected in common to the16 reset transistors 1303.

The drain of each amplifying transistor 1304 is connected with the Vddinterconnection 1310 to which power supply voltage is supplied. Also,the source of each amplifying transistor 1304 is connected with thedrain of each corresponding selecting transistor 1305. The gate of eachselecting transistor is connected with a decoder interconnection 1308 towhich selection pulses are supplied. In the present embodiment, thedecoder interconnection 1308 is provided independently to each of the 16selecting transistors 1305. Then, the source of each selectingtransistor 1305 is connected with a common output interconnection 1309.A load current source 1311 supplies current to the outputinterconnection 1309. That is, the output interconnection 1309 for theselecting transistors 1305 is formed by a source follower. Note that theload current source 1311 may be provided on the imaging chip 1113 sideor on the signal processing chip 1111 side.

Here, a flow from the start of electrical charge accumulation to pixeloutput after the end of the accumulation will be explained. When resetpulses are applied to the reset transistor 1303 through the resetinterconnection 1306, and simultaneously transfer pulses are applied tothe transfer transistor 1302 through the TX interconnection 1307,potential of the PD 1104 and the floating diffusion FD is reset.

When the application of the transfer pulses is stopped, the PD 1104converts received incident light into electrical charges, which are thenaccumulated. Thereafter, when transfer pulses are applied again in astate where reset pulses are not being applied, accumulated electricalcharges are transferred to the floating diffusion FD, and the potentialof the floating diffusion FD changes from reset potential to signalpotential after electrical charge accumulation. Then, when selectionpulses are applied to the selecting transistor 1305 through the decoderinterconnection 1308, variation in the signal potential of the floatingdiffusion FD is transmitted to the output interconnection 1309 via theamplifying transistor 1304 and the selecting transistor 1305. Thereby,pixel signals corresponding to the reset potential and the signalpotential are output from the unit pixel to the output interconnection1309.

As illustrated, in the present embodiment, the reset interconnection1306 and the TX interconnection 1307 are common to the 16 pixels thatform the unit group 1131. That is, the reset pulses and the transferpulses are, respectively, applied simultaneously to all the 16 pixels.Accordingly, all the pixels that form the unit group 1131 startelectrical charge accumulation at the same timing, and end electricalcharge accumulation at the same timing. Note that however pixel signalsthat correspond to accumulated electrical charges are output selectivelyto the output interconnection 1309 upon sequential application ofselection pulses to the respective selecting transistors 1305. Also, thereset interconnection 1306, the TX interconnection 1307, and the outputinterconnection 1309 are provided separately for each unit group 1131.

By configuring a circuit on the basis of the unit group 1131 in thismanner, an electrical charge accumulation period can be controlled foreach unit group 1131. In other words, adjacent unit groups 1131 can becaused to output pixel signals for different electrical chargeaccumulation periods. Furthermore, by causing one unit group 1131 torepeat electrical charge accumulation several times and output a pixelsignal at each time while another unit group 1131 is caused to performelectrical charge accumulation once, these unit groups 1131 can becaused to output respective frames for a motion image at different framerates.

FIG. 25 is a block diagram illustrating a configuration of an imagingdevice according to the present embodiment. An imaging device 1500includes an imaging lens 1520 as an imaging optical system, and theimaging lens 1520 guides a subject luminous flux that is incident alongan optical axis OA to the imaging element 1100. The imaging lens 1520may be a replaceable lens that can be attached/detached to and from theimaging device 1500. The imaging device 1500 includes, mainly, theimaging element 1100, a system control unit 1501, a drive unit 1502, aphotometry unit 1503, a work memory 1504, a recording unit 1505, and adisplay unit 1506.

The imaging lens 1520 is configured with a plurality of optical lensgroups, and forms an image of a subject luminous flux from a scene nearits focal plane. Note that, in FIG. 25, the imaging lens 1520 isrepresentatively shown with a single virtual lens that is placed nearthe pupil. The drive unit 1502 is a control circuit that executeselectrical charge accumulation control such as timing control and areacontrol on the imaging element 1100 according to instructions from thesystem control unit 1501.

The imaging element 1100 passes pixel signals over to an imageprocessing unit 1511 of the system control unit 1501. The imageprocessing unit 1511 performs various types of image processing by usingthe work memory 1504 as a workspace, and generates image data. Forexample, when image data in a JPEG file format is generated, compressionprocesses are executed after color video signals are generated fromsignals obtained from Bayer arrays. The generated image data is recordedin the recording unit 1505 and converted into display signals, and isdisplayed on the display unit 1506 for a preset period of time.

The photometry unit 1503 detects luminance distribution of a scene priorto an imaging sequence for generating image data. The photometry unit1503 includes an AE sensor of approximately one million pixels, forexample. A computing unit 1512 of the system control unit 1501calculates luminance of respective areas within a scene, upon receivingan output of the photometry unit 1503. The computing unit 1512 decides ashutter speed, a diaphragm value, and an ISO speed according to thecalculated luminance distribution. The imaging element 1100 may doubleas the photometry unit 1503. Note that the computing unit 1512 executesvarious types of computation for operating the imaging device 1500.

The drive unit 1502 may be partially or entirely mounted on the imagingchip 1113, or partially or entirely mounted on the signal processingchip 1111. The system control unit 1501 may be partially mounted on theimaging chip 1113 or the signal processing chip 1111.

FIG. 26 is a block diagram that illustrates a specific configuration ofthe signal processing chip 1111 as one example.

The signal processing chip 1111 serves functions of the drive unit 1502.

The signal processing chip 1111 includes a sensor control unit 1441, ablock control unit 1442, a synchronization control unit 1443, a signalcontrol unit 1444, an individual circuit unit 1450A, and the like thatserve divided control functions, and a drive control unit 1420 thatperforms overall control on the respective control units. The signalprocessing chip 1111 further includes an I/F circuit 1418 between thedrive control unit 1420 and the system control unit 1501 of the mainbody of the imaging device 1500. These sensor control unit 1441, blockcontrol unit 1442, synchronization control unit 1443, signal controlunit 1444, and drive control unit 1420 are each provided to each of thesignal processing chips 1111.

On the other hand, the individual circuit units 1450A, 1450B, 1450C,1450D, 1450E are provided to the unit groups 1131A, 1131B, 1131C, 1131D,1131E, respectively. Because the individual circuit units 1450A, 1450B,1450C, 1450D, 1450E have the same configuration, the individual circuitunit 1450A is explained below. The individual circuit unit 1450Aincludes a CDS circuit 1410, a multiplexer 1411, an A/D convertingcircuit 1412, a de-multiplexer 1413, a pixel memory 1414, and anarithmetic circuit 1415. The arithmetic circuit 1415 transmits andreceives signals to and from the system control unit 1501 via the I/Fcircuit 1418.

The individual circuit unit 1450A is preferably disposed in an areasuperimposed on an area where pixels of the corresponding unit group1131A are disposed. Thereby, the individual circuit unit 1450A can beprovided to each of a plurality of the unit groups 1131A withoutincreasing the size of each chip in the plane direction.

The drive control unit 1420 refers to a timing memory 1430, convertsinstructions from the system control unit 1501 into control signals thatcan be executed by the respective control units, and passes them over tothe respective control units. In particular, when each of the unitgroups 1131A and the like is controlled by using separate controlparameters, the drive control unit 1420 passes the control parameter toeach control unit together with information that identifies the unitgroup 1131A. When controlling acquisition of a single image, once thedrive control unit 1420 receives a signal of an imaging instruction fromthe system control unit 1501, the drive control unit 1420 thereafter isable to complete accumulation control without receiving an instructionabout control on each pixel from the system control unit 1501 each time.

The sensor control unit 1441 performs transmission control on controlpulses that are to be transmitted to the imaging chip 1113 and relate toelectrical charge accumulation and electrical charge readout of eachpixel. Specifically, the sensor control unit 1441 controls the start andend of electrical charge accumulation by transmitting reset pulses andtransfer pulses to target pixels, and causes pixel signals to be outputto the output interconnection 1309 by transmitting selection pulses toreadout pixels.

The block control unit 1442 executes transmission of specifying pulsesthat are to be transmitted to the imaging chip 1113 and specify a unitgroup 1131 to be controlled. Transfer pulses and reset pulses that eachpixel receives via the TX interconnection 1307 and the resetinterconnection 1306 are logical AND of each pulse transmitted by thesensor control unit 1441 and specifying pulses transmitted by the blockcontrol unit 1442. In this manner, each area can be controlled as ablock that is separate from other areas.

The synchronization control unit 1443 transmits a synchronization signalto the imaging chip 1113. Each pulse becomes active in the imaging chip1113 in synchronization with the synchronization signal. For example, byadjusting the synchronization signal, random control, thinning control,and the like only on particular pixels among pixels belonging to thesame unit group 1131A can be realized.

The signal control unit 1444 mainly performs timing control on the A/Dconverting circuit 1412. Pixel signals output via the outputinterconnection 1309 are input to the A/D converting circuit 1412through the CDS circuit 1410 and the multiplexer 1411. The CDS circuit1410 eliminates noises from pixel signals.

The A/D converting circuit 1412 is controlled by the signal control unit1444 to convert the input pixel signals into digital signals. The pixelsignals converted into the digital signals are passed over to thede-multiplexer 1413, and are stored as a pixel value of digital data inthe pixel memory 1414 corresponding to each pixel.

A data transfer interface that transmits pixel signals according to adelivery request is provided to the pixel memory 1414. The data transferinterface is connected with a data transfer line that connects with theimage processing unit 1511. The data transfer line is configured forexample with a data bus among bus lines. In this case, a deliveryrequest from the system control unit 1501 to the drive control unit 1420is executed by addressing that utilizes an address bus.

Transmission of pixel signals by the data transfer interface is notlimited to an addressing system, but may adopt various systems. Forexample, at the time of data transfer, a double data rate system inwhich both rising and falling of a clock signal used for synchronizationof each circuit are utilized to perform processing may be adopted. Also,a burst transfer system of transferring data at once by partiallyomitting procedures such as addressing, and attempting speed up may beadopted. Also, a bus system of using lines that connect a control unit,a memory unit, and an input/output unit in parallel, and a serial systemof transferring data in series on a bit by bit basis may be adopted incombination.

With this configuration, because the image processing unit 1511 canreceive only necessary pixel signals, the image processing unit 1511 cancomplete image processing at high speed particularly when forming a lowresolution image. Also, because when the arithmetic circuit 1415 iscaused to execute an integration process, the image processing unit 1511does not have to execute the integration process, speeding up of theimage processing may be attempted by functional division and parallelprocessing.

The signal processing chip 1111 has the timing memory 1430 that isformed with a flash RAM and the like. The timing memory 1430 storescontrol parameters such as information on the number of times ofaccumulation about how many times electrical charge accumulation is tobe repeated for which unit group 1131A or the like, in association withinformation that identifies the unit group 1131A or the like. Any of thecontrol parameters is calculated by the arithmetic circuit 1415 of theindividual circuit unit 1450A or the like, and stored in theabove-described timing memory 1430.

The drive control unit 1420 which executes electrical chargeaccumulation control on the imaging chip 1113 further refers to thetiming memory 1430 in execution of readout control. For example, thedrive control unit 1420 refers to information on the number of times ofaccumulation of each unit group 1131 to store a pixel signal output fromthe de-multiplexer 1413 in a corresponding address of the pixel memory1414.

The drive control unit 1420 reads out a target pixel signal from thepixel memory 1414 according to a delivery request from the systemcontrol unit 1501, and passes it over to the image processing unit 1511.The pixel memory 1414 has a memory space that can store pixel signalscorresponding to the maximum number of times of integration about eachpixel, and stores, as pixel values, their respective pixel signalscorresponding to the number of times of accumulation executed. Forexample, because when electrical charge accumulation is repeated fourtimes in a block, pixels included in the block output pixel signals thatcorrespond to the four times, a memory space in the pixel memory 1414for each pixel stores four pixel values. When having received, from thesystem control unit 1501, a delivery request that requests a pixelsignal of a particular pixel, the drive control unit 1420 specifies anaddress of the particular pixel on the pixel memory 1414, reads out allthe stored pixel signals, and passes them over to the image processingunit 1511. For example when four pixel values are stored, all the fourpixel values are sequentially passed over, and when only one pixel valueis stored, the pixel value is passed over.

The drive control unit 1420 can read out a pixel signal stored in thepixel memory 1414, pass it to the arithmetic circuit 1415, and cause thearithmetic circuit 1415 to execute the above-described integrationprocess. The pixel signal having been subjected to the integrationprocess is stored in a target pixel address of the pixel memory 1414.The target pixel address may be provided adjacent to an address spacebefore the integration process, or may be the same address so that apixel signal is written over the pixel signal before the integrationprocess. Also, a dedicated space that collectively stores pixel valuesof respective pixels after the integration process may be provided. Whenhaving received, from the system control unit 1501, a delivery requestthat requests a pixel signal of a particular pixel, the drive controlunit 1420 can pass the pixel signal after the integration process overto the image processing unit 1511 depending on the form of the deliveryrequest. Of course, pixel signals before and after the integrationprocess may be passed over together.

As described above, the output interconnection 1309 is providedcorresponding to each of the unit groups 1131. Because the imagingelement 1100 is formed by layering the imaging chip 1113, the signalprocessing chip 1111, and the memory chip 1112, the outputinterconnection 1309 can be routed without increasing the size of eachchip in the plane direction by using inter-chip electrical connectionsthat use bumps 1109 for the interconnection. Similarly, for signal linesfrom each control unit to a unit group, the interconnection can berouted without increasing the size of each chip in the plane directionby using inter-chip electrical connections that use the bumps 1109.

FIG. 27 shows one example of functional blocks of the arithmetic circuit1415. The arithmetic circuit 1415 computes an evaluation value by usinga pixel signal stored in the pixel memory 1414 of the individual circuitunit 1450A, and based on the evaluation value, outputs a controlparameter for controlling exposure or readout of the corresponding unitgroup 1131A. In the example illustrated in FIG. 27, the arithmeticcircuit 1415 calculates a frame rate to be applied to the pixel unitgroup 1131 based on a difference between averages of pixel signals ofthe unit group 1131A in a time-series.

The arithmetic circuit 1415 in FIG. 27 has an average calculating unit1452, an average storage unit 1454, a difference calculating unit 1456,and a frame rate calculating unit 1458. The average calculating unit1452 calculates an average value Ag by calculating a simple average of Gpixel signals of each pixel of the unit group 1131A that are stored inthe pixel memory 1414. In this case, the average calculating unit 1452calculates the average value Ag of a frame at the time at time intervalsthat correspond to a predetermined frame rate.

In the above-described example, a single value of the average value Agis calculated for each unit group 1131A, and is stored in the averagestorage unit 1454. Because the difference between the average values Agof preceding and following frames is calculated, a memory space thatstores at least two values is provided in the average storage unit 1454.

The difference calculating unit 1456 calculates a difference d betweenthe average value Ag of a latest frame that is stored in the averagestorage unit 1454, and the average value Ag of a temporally precedingframe. The difference may be output as an absolute value.

The frame rate calculating unit 1458 calculates a frame rate f bycomparing the difference d calculated by the difference calculating unit1456 with a predetermined reference value d0 or the like. Here, forexample, a table in which larger inter-frame differences d areassociated with higher frame rates f is stored in the frame ratecalculating unit 1458.

The frame rate calculating unit 1458 outputs the calculated frame rate fto the drive control unit 1420. Instead of or in addition to this, theframe rate calculating unit 1458 may directly write the frame rate f inthe timing memory 1430.

FIG. 28 illustrates one example of correspondence between inter-framedifferences d, and frame rates f. In FIG. 28, differences between frameshave two reference values d0, d1, and corresponding to the values, threelevels of frame rates f0, f1, f2 are provided.

When the difference d between frames is equal to or smaller than thelower reference value d0, the frame rate calculating unit 1458 outputsthe lowest frame rate f0 as the frame rate f to be applied to the unitgroup 1131A. Also, when the difference d between frames is between thereference value d0 and the higher reference value d1, the frame ratecalculating unit 1458 outputs the intermediate frame rate f1. When thedifference d between frames is larger than the reference value d1, theframe rate calculating unit 1458 output the highest frame rate f2.

Here, time intervals at which the arithmetic circuit 1415 performs theabove-described series of computation are preferably set to (1/f0) thatcorresponds to intervals between frames of the lowest frame rate f0.Thereby, irrespective of the frame rate at which driving is performed atthe time, a next frame rate can be calculated at timing which is thesame for a plurality of the unit groups 1131A, 1131B, and the like.Also, even when driving is performed at the lowest frame rate f0, a newframe rate f can be calculated based on frames that are different fromthose used in previous calculation.

FIGS. 29 and 30 each illustrate one example of images imaged by animaging element. Note that grid lines in an image 1170 and an image 1178indicate boundaries of the unit groups 1131, but the number of the unitgroups 1131 is merely an example, and is not limited to the number shownin the figures. Also, the unit group 1131A or the like is simply denotedwith “A” or the like. Unit groups including a main subject 1171 areindicated with thick lines.

It is assumed that the imaging element 1100 for example acquires theimage 1170 in FIG. 29 and the image 1178 in FIG. 30 as temporallysequential images. Paying attention to the unit group 1131A in thefigures, the unit group 1131A includes the main subject 1171 not in theimage 1170 of a former frame, but in the image 1178 of a latter frame.Accordingly, the difference d of the average values Ag of the unit group1131A between the image 1170 and the image 1178 that is calculated bythe average calculating unit 1452 becomes large.

Thereby, the frame rate calculating unit 1458 calculates the frame ratef of the unit group 1131A of and after the image 1178 to be high basedon the correspondence in FIG. 28. Accordingly, the drive control unit1420 drives each pixel of the unit group 1131A of and after the image1178 at the high frame rate f2 or the like. Therefore, the drive controlunit 1420 can acquire pixel signals of a subject whose motion is largebetween temporally preceding and following frames, at the high framerate f2 or the like.

Electrical charge accumulation can be performed multiple times in unitgroups 1131 which are driven at the high frame rate f2 while electricalcharge accumulation is performed once in unit groups 1131 which aredriven at the low frame rate f0. Accordingly, the number of bits at thetime when pixel signals of the unit groups 1131 driven at the high framerate f2 or the like are digitized can be made larger than that for theunit groups 1131 driven at the low frame rate f0. Thereby, an image witha high gradation can be generated from the unit groups 1131 driven atthe high frame rate f2 or the like.

Instead of increasing the number of bits for digitization, an S/N ratiomay be improved by performing image processing of averaging imagesacquired at the high frame rate f2 or the like. In this case, imagesignals that correspond to multiple times, for example four times, areobtained from the unit groups 1131 driven at the high frame rate f2, andstored in the pixel memory 1414, while electrical charge accumulation isperformed once in the unit groups 1131 driven at the low frame rate f0.The arithmetic circuit 1415 reads out, from the pixel memory 1414, aplurality of pixel signals obtained for each pixel of the unit groups1131 controlled at the high frame rate f2, and averages them forrespective pixels. Thereby, random noises of each pixel of the unitgroups 1131 are reduced, and an S/N ratio can be improved.

In the above-described manner, the frame rate f can be calculatedpromptly and with less power, as compared with calculating the framerate f of each unit group 1131A or the like after acquiring, by theimage processing unit 1511 in the subsequent step, pixel signals of theentire image 1170 or the like to estimate a main subject. Also, even ina case that a pixel in any of the unit groups 1131 experiences failuresof interconnection, processing circuits, and the like, the frame rate fcan be calculated promptly and with less electrical power for other unitgroups 1131.

Note that the average calculating unit 1452 in FIG. 27 averages pixelsignals of G pixels in the corresponding unit group 1131A. Instead ofthis, the average calculating unit 1452 may calculate an average thatreflects pixel signals of R pixels and B pixels. Also, the averagecalculating unit 1452 may calculate an average of G pixels, an averageof R pixels, and an average of B pixels. In this case, the frame ratecalculating unit 1458 may calculate the frame rate f based on acondition whether any of a difference between averages of G pixels, adifference between averages of R pixels, and a difference betweenaverages of B pixels exceeds a threshold, or other conditions.Furthermore, determination may be made based on a result of adding anaverage of G pixels, an average of R pixels, and an average of B pixelsat a predetermined ratio. Also, the average values may be calculated asaverage values of a partial area placed within a unit group.

Also, the average calculating unit 1452 may acquire the average value Agof the unit groups 1131B, 1131C, 1131D, 1131E, and the like in theperiphery of the unit group 1131A as illustrated in FIG. 29, etc. fromthe arithmetic circuit 1415 of another individual circuit unit 1450 andthe like, and take it into consideration about the average value Ag ofthe unit group 1131A For example, the average values may beweight-averaged. Instead of acquiring the average value Ag of the unitgroups 1131B, 1131C, 1131D, 1131E, and the like in the periphery of theunit group 1131A from the other arithmetic circuit 1415 or the like, theaverage calculating unit 1452 itself may read out pixel signals from thepixel memory 1414 of the other individual circuit units 1450B and thelike to calculate the average value Ag.

Also, although in the example in FIG. 28, there are the two referencevalues of differences, and the three levels of frame rates, the numberof the reference values of differences and the number of the levels offrame rates are not limited thereto.

FIG. 31 shows one example of functional blocks of another arithmeticcircuit 1416. In the example illustrated in FIG. 31, the arithmeticcircuit 1416 calculates a thinning rate to be applied to the pixel unitgroup 1131A based on contrast of pixel signals of the unit group 1131A.

The arithmetic circuit 1416 in FIG. 31 has a high-frequency componentcalculating unit 1460, a summation calculating unit 1462, and a thinningrate calculating unit 1464. The high-frequency component calculatingunit 1460 reads out a G pixel signal of each pixel in the unit group1131A stored in the pixel memory 1414, and performs high-pass filterprocessing based on its two-dimensional array to extract a spatialhigh-frequency component Gh. Similarly, the high-frequency componentcalculating unit 1460 calculates a high-frequency component Rh of Rpixels and a high-frequency component Bh of B pixels.

The summation calculating unit 1462 calculates a summation of absolutevalues of the above-described high-frequency components Gh, Rh, Bh. Thethinning rate calculating unit 1464 calculates a thinning rate at whichpixels included in the unit group 1131A are thinned and read out, basedon the above-described summation. In this case, a table in which largersummations are associated with lower thinning rates is preferablyprestored in the thinning rate calculating unit 1464. For example, inplace of the correspondence in FIG. 28, reference values of summationsand thinning rates are associated.

For example, a single reference value of summations is set, and when asummation is larger than the reference value, thinning is not performedand all the pixels are read out, and when the summation is smaller thanthe reference value, the thinning rate of 0.5 is calculated. Thethinning rate calculating unit 1464 outputs the calculated thinning rateto the drive control unit 1420. Instead of or in addition to this, thethinning rate calculating unit 1464 may directly write the thinning ratein the timing memory 1430.

The drive control unit 1420 causes output of image signals to beexecuted, by thinning pixels included in a corresponding unit group 1131at the above-described thinning rate calculated by the thinning ratecalculating unit 1464. In this case, the drive control unit 1420 obtainspixel signals at different thinning rates by separately driving a set ofthe reset transistors 1303, the transfer transistors 1302, and theselecting transistors 1305 of the unit groups 1131 for which thethinning rate of 0.5 has been calculated, and a set of the resettransistors 1303, the transfer transistors 1302, and the selectingtransistors 1305 of the unit groups 1131 for which the thinning rate of0 has been calculated.

Thereby, the signal amount can be reduced for unit groups 1131 thatcorrespond to a low contrast area while keeping the resolution of unitgroups 1131 that correspond to a high contrast area high Furthermore, inthis case, a thinning rate can be calculated promptly and with lesselectrical power, as compared with calculating a thinning rate by theimage processing unit 1511 in the subsequent step. Also, even in a casethat a pixel in any of the unit groups 1131 experiences failures ofinterconnection, processing circuits, and the like, a thinning rate canbe calculated promptly and with less electrical power for other unitgroups 1131.

FIG. 32 illustrates an example of pixels 1188 to be read out group atthe thinning rate of 0.5 in one unit group. In the example illustratedin FIG. 32, when a unit group 1132 is a Bayer array, the pixels 1188 tobe read out and pixels not to be read out are set for every other Bayerarray, that is, every two pixels alternately in the vertical direction.Thereby, thinned readout can be performed without losing a colorbalance.

Although rows to be read out are thinned in the example in FIG. 32,instead of this, columns to be read out may be thinned Furthermore, thehigh-frequency component calculating unit 1460 may extract ahigh-wavelength component each in the column direction and the rowdirection, and the thinning rate calculating unit 1464 may calculate athinning rate in the column direction and a thinning rate in the rowdirection.

In the configurations in FIGS. 31 and 32, the thinning rate calculatingunit 1464 calculates a thinning rate of a corresponding pixel group.Instead of this, the number of pixels for adding pixel signals of theadjacent same color pixels may be calculated. For example, when asummation calculated by the summation calculating unit 1462 is equal toor larger than a reference value, the number of rows becomes 1, that is,pixel signals are output without addition between the adjacent samecolor pixels, and when the summation is smaller than the referencevalue, the number of rows becomes larger, for example two, and pixelsignals are output by performing addition between two adjacent rows ofthe same color pixels in the same columns.

Thereby, similar to FIG. 32, the overall signal amount can be reducedwhile keeping the resolution of a high contrast area high. Also, insteadof adding the same color pixel signals in adjacent rows, the same colorpixel signals in adjacent columns may be added. Furthermore, in theabove-described addition, a process of calculating an average bydividing the sum value by the number of added rows or columns may beincluded. Also, the same color pixel signals in adjacent rows andcolumns may be added.

Note that high-wavelength components Rh, Gh, Bh for each of R pixels, Gpixels, and B pixels are used in the above-described high-frequencycomponent calculating unit 1460 and the like. Instead of this,high-frequency components may be determined by using luminancecomponents calculated from R pixels, G pixels, and B pixels. In thiscase, the high-frequency components may be determined after adjustinggains among the luminance components of R pixels, G pixels, and Bpixels.

Also, the summation calculating unit 1462 may acquire high-frequencycomponents of the unit groups 1131B, 1131C, 1131D, 1131E, and the likein the periphery of the unit group 1131A as illustrated in FIG. 29 andthe like from the arithmetic circuits 1416 of the other individualcircuit units 1450B and the like, and take it into consideration about ahigh-frequency component of the unit group 1131A. For example, theaverage values may be weight-averaged. Instead of acquiring the averagevalues Ag of the unit groups 1131B, 1131C, 1131D, 1131E, and the like inthe periphery of the unit group 1131A from the other arithmetic circuits1416 and the like, the summation calculating unit 1462 itself may readout pixel signals from the pixel memories 1414 of the other individualcircuit units 1450B and the like to calculate the high-frequencycomponent.

Also, the number of bits for digitization of pixel signals may be madelarger for pixel signal of unit groups that exceed thresholds in theframe rate calculating unit 1458 and the thinning rate calculating unit1464 than those for unit groups that do not exceed the thresholds. Forexample, the A/D converting circuit 1412 may perform digitization with alarger number of bits for the same one-time accumulation, according toan instruction from the drive unit 1502.

FIG. 33 shows still another example of functional blocks of anarithmetic circuit 1417. The arithmetic circuit 1417 has a self-averagecalculating unit 1472, an adjacent average calculating unit 1470, a gaincalculating unit 1474, and a correcting unit 1476.

The self-average calculating unit 1472 calculates a simple average of Gpixel signals of each pixel in the unit group 1131A that are stored inthe pixel memory 1414 to calculate the average value Ag. Similarly, theself-average calculating unit 1472 calculates respective simple averagesof R pixel signals and B pixel signals of each pixel in the unit group1131A that are stored in the pixel memory 1414 to calculate the averagevalues Ar, Ab. Furthermore, the self-average calculating unit 1472outputs the average values Ag, Ar, Ab of the unit group 1131A to theadjacent average calculating units 1470 of the peripheral unit groups1131B and the like.

The adjacent average calculating unit 1470 acquires the average valuesAg, Ar, Ab from the self-average calculating unit 1472 corresponding tothe other unit groups 1131B, 1131C, 1131D, 1131E adjacent to the unitgroup 1131A, and calculates their weighted average. The gain calculatingunit 1474 calculates, for each of RGB, a weighted average of the averagevalues Ag, Ar, Ab calculated by the self-average calculating unit 1472and the average values Ag, Ar, Ab calculated by the adjacent averagecalculating unit 1470, and based on their ratio, calculates gains of Rpixel signals and B pixel signals in relation to G pixel signals. Inthis case, for example, weighted-averaging is performed by using theweighting of 4/8 for the average value of the unit group 1131A and theweighting of ⅛ for the average value of the adjacent unit group 1131Band the like.

The gains of R pixel signals and B pixel signals are transmitted asadditional information to the system control unit 1501 via the I/Fcircuit 1418. Note that instead of acquiring the average values Ag ofthe unit groups 1131B, 1131C, 1131D, 1131E, and the like in theperiphery of the unit group 1131A from the arithmetic circuits 1417 ofthe other individual circuit units 1450B and the like, the adjacentaverage calculating unit 1470 itself may read out pixel signals from thepixel memories 1414 of the other individual circuit units 1450B and thelike to calculate the average values Ag and the like.

The correcting unit 1476 corrects R pixel signals and B pixel signals byusing the gains calculated by the gain calculating unit 1474, and writethem in the pixel memory 1414. In this case, the correcting unit 1476multiplies individual R pixel signals by the gain for the R pixelsignals, and multiplies individual B pixel signals by the gain for the Bpixel signals. Note that the correcting unit 1476 may further correctthe gains by acquiring feedback information from the system control unit1501.

FIG. 34 schematically illustrates a relationship between gains and pixelsignals. Operations of calculating gains and correcting pixel signalsare preferably performed for example for every frame at the frame ratef0, that is, every (1/f0) seconds. As illustrated in FIG. 34, a gain ofR pixel signals and a gain of B pixel signals are calculated every (1/M)seconds, and an output value of the R pixel signals and an output valueof the B pixel signals are corrected. In the above-described manner,pixel signals can be corrected promptly and with less power, as comparedwith calculating a gain and correcting the pixel signals by the imageprocessing unit 1511 in the subsequent step.

In the above-described embodiment, the sensor control unit 1441, theblock control unit 1442, the synchronization control unit 1443, thesignal control unit 1444, and the drive control unit 1420 are eachprovided to each of the signal processing chips 1111, and the individualcircuit units 1450A, 1450B, 1450C, 1450D, 1450E are provided to the unitgroups 1131A, 1131B, 1131C, 1131D, 1131E, respectively. Instead of this,a plurality of the sensor control unit 1441, the block control unit1442, the synchronization control unit 1443, the signal control unit1444, and the drive control unit 1420 may be provided to each of thesignal processing chip 1111, and each of them may share the role ofcontrolling a plurality of unit groups 1131.

Also, the individual circuit unit 1450A or the like may be provided eachto a plurality of unit groups 1131, and may be shared by the pluralityof unit groups 1131. The individual circuit unit 1450A or the like maybe provided each to pixels. That is, in the above-described embodiment,the unit group 1131 may include a single pixel.

FIG. 35 is a sectional view of another imaging element 2100 according tothe present embodiment. The imaging element 2100 includes an imagingchip 2113 that outputs a pixel signal corresponding to incident light, asignal processing chip 2111 that processes the pixel signal, and amemory chip 2112 that stores the pixel signal. These imaging chip 2113,signal processing chip 2111, and memory chip 2112 are layered, and areelectrically connected with each other via conductive bumps 2109, suchas Cu.

Note that, as illustrated, incident light is incident mainly in the Zaxis positive direction that is indicated with an outlined arrow. Inthis specification, the surface of the imaging chip 2113 on a side onwhich the incident light is incident is called a backside. Also, asindicated with coordinate axes, the leftward direction on the figurethat is orthogonal to the Z axis is referred to as the X axis positivedirection, and the front side direction in the figure that is orthogonalto the Z and X axes is referred to as the Y axis positive direction. Inseveral figures mentioned below, the coordinate axes are displayed suchthat the orientation of each figure can be known on the basis of thecoordinate axes in FIG. 35.

One example of the imaging chip 2113 is a backside illuminating type MOSimage sensor. A PD layer 2106 is disposed on a backside of aninterconnection layer 2108. The PD layer 2106 has a plurality of PDs(photo diodes) 2104 that are two-dimensionally disposed, accumulateelectrical charges according to incident light, and generate pixelsignals according to the accumulated electrical charges, and transistors2105 provided corresponding to the PDs 2104.

Color filters 2102 are provided on the incident light incidence side ofthe PD layer 2106 via a passivation film 2103. There is a plurality oftypes of the color filters 2102 that allow passage of mutually differentwavelength ranges, and the color filters 2102 are arrayed particularlycorresponding to the respective PDs 2104. The arrays of the colorfilters 2102 are described below. A set of the color filter 2102, the PD2104, and the transistor 2105 forms one pixel.

A microlens 2101 is provided, corresponding to each pixel, on theincident light incidence side of the color filter 2102. The microlens2101 condenses incident light toward the corresponding PD 2104.

The interconnection layer 2108 has interconnections 2107 that transmit apixel signal from the PD layer 2106 to the signal processing chip 2111.The interconnection 2107 may be a multilayer, and may be provided with apassive element and an active element.

A plurality of the bumps 2109 is disposed on a surface of theinterconnection layer 2108. The plurality of bumps 2109 are aligned witha plurality of the bumps 2109 that are provided on the opposing surfaceof the signal processing chip 2111, and, for example, the imaging chip2113 and the signal processing chip 2111 are pressed against each other;thereby, the aligned bumps 2109 are bonded and electrically connectedwith each other.

Similarly, a plurality of the bumps 2109 are disposed on the mutuallyopposing surfaces of the signal processing chip 2111 and the memory chip2112. These bumps 2109 are aligned with each other, and, for example,the signal processing chip 2111 and the memory chip 2112 are pressedagainst each other; thereby, the aligned bumps 2109 are bonded andelectrically connected with each other.

Note that bonding between the bumps 2109 is not limited to Cu bumpbonding by solid phase diffusion, but microbump joining by soldermelting may be adopted. Also, approximately one bump 2109 may beprovided, for example, for each pixel block described below.Accordingly, the size of the bumps 2109 may be larger than the pitch ofthe PDs 2104. Also, in a peripheral area other than an imaging areawhere pixels are arrayed, a bump that is larger than the bumps 2109corresponding to the imaging area may also be provided.

The signal processing chip 2111 has a TSV (through-silicon via) 2110that connects circuits that are provided on a frontside and a backside,respectively. The TSV 2110 is preferably provided in the peripheralarea. Also, the TSV 2110 may be provided also in the peripheral area ofthe imaging chip 2113, and the memory chip 2112.

FIG. 36 is a diagram for explaining a pixel array and a pixel block 2131of the imaging chip 2113. FIG. 36 shows a state of the imaging chip 2113as observed from the backside A matrix of a plurality of pixels isarrayed in an imaging area 2700. The imaging area 2700 has a pluralityof the pixel blocks 2131 that are formed by dividing a plurality ofpixels in the row and column directions. Each pixel block 2131 has m (npixels in the row and column directions. Here, m and n are integers thatare equal to or larger than two. Also, the row and column directionsrefer to two different directions in the plane of the imaging area 2700,and may not necessarily be orthogonal to each other. In FIG. 36,adjacent four pixels (four pixels, 16 pixels, form one pixel block 2131.Grid lines in the figure show the concept that adjacent pixels aregrouped to form the pixel block 2131. The number of pixels that form thepixel block 2131 is not limited thereto, but may be approximately 1000,for example thirty two pixels (sixty four pixels, or more or less.

As illustrated in the partially enlarged view of the imaging area 2700,the pixel block 2131 includes, within its upper left, upper right, lowerleft, and lower right portions, four so-called Bayer arrays eachincluding four pixels including green pixels Gb, Gr, a blue pixel B, anda red pixel R. The green pixels have green filters as the color filters2102, and receive light in the green wavelength band of incident light.Similarly, the blue pixel has a blue filter as the color filter 2102,and receives light in the blue wavelength band, and the red pixel has ared filter as the color filter 2102, and receives light in the redwavelength band.

In the present embodiment, at least one pixel block is selected fromamong a plurality of pixel blocks 2131, and pixels included in eachpixel block are controlled with control parameters that are differentfrom those for other pixel blocks. Examples of the control parametersinclude a frame rate, a thinning rate, the number of added rows whosepixel signals are added, a period or the number of times of accumulatingelectrical charges, the number of bits for digitization, and the like.Furthermore, the control parameters may be parameters in imageprocessing performed after acquiring image signals from a pixel. Theframe rate refers to a cycle of generating pixel signals. Note that inthis specification, the frame rate may refer to frame rates of therespective pixel blocks 2131. For example, a reference frame rate and ahigh frame rate refer to frame rates of the respective pixel blocks 2131

FIG. 37 is a schematic that corresponds to the pixel block 2131 of theimaging chip 2113. In the figure, a rectangle that is indicated withdotted lines representatively represents a circuit that corresponds toone pixel. Note that at least a part of each transistor explained belowcorresponds to the transistor 2105 in FIG. 35.

Although in FIG. 37, the pixel block 2131 formed with 16 pixels isillustrated, the number of pixels of the pixel block 2131 is not limitedthereto. The 16 PDs 2104 that correspond to respective pixels areconnected with respective transfer transistors 2302, and the gate ofeach transfer transistor 2302 is connected with a TX interconnection2307 to which transfer pulses are supplied. In the example illustratedin FIG. 37, the TX interconnection 2307 is connected in common to the 16transfer transistors 2302.

The drain of each transfer transistor 2302 is connected with the sourceof each corresponding reset transistor 2303, and also a so-calledfloating diffusion FD between the drain of the transfer transistor 2302and the source of the reset transistor 2303 is connected with the gateof an amplifying transistor 2304. The drain of the reset transistor 2303is connected with a Vdd interconnection 2310 to which power supplyvoltage is supplied, and its gate is connected with a resetinterconnection 2306 to which reset pulses are supplied. In the exampleillustrated in FIG. 37, the reset interconnection 2306 is connected incommon to the 16 reset transistors 2303.

The drain of each amplifying transistor 2304 is connected with the Vddinterconnection 2310 to which power supply voltage is supplied. Also,the source of each amplifying transistor 2304 is connected with thedrain of each corresponding selecting transistor 2305. The gate of eachselecting transistor is connected with a decoder interconnection 2308 towhich selection pulses are supplied. In the example illustrated in FIG.37, the decoder interconnection 2308 is provided independently to eachof the 16 selecting transistors 2305. Then, the source of each selectingtransistor 2305 is connected with a common output interconnection 2309.A load current source 2311 supplies current to the outputinterconnection 2309. That is, the output interconnection 2309 for theselecting transistors 2305 is formed by a source follower. Note that theload current source 2311 may be provided on the imaging chip 2113 sideor on the signal processing chip 2111 side.

Here, a flow from the start of electrical charge accumulation to pixeloutput after the end of the accumulation will be explained. When resetpulses are applied to the reset transistor 2303 through the resetinterconnection 2306, and simultaneously transfer pulses are applied tothe transfer transistor 2302 through the TX interconnection 2307,potential of the PD 2104 and the floating diffusion FD is reset.

When the application of the transfer pulses is stopped, the PD 2104converts received incident light into electrical charges, which are thenaccumulated. Thereafter, when transfer pulses are applied again in astate where reset pulses are not being applied, accumulated electricalcharges are transferred to the floating diffusion FD, and the potentialof the floating diffusion FD changes from reset potential to signalpotential after electrical charge accumulation. Then, when selectionpulses are applied to the selecting transistor 2305 through the decoderinterconnection 2308, variation in the signal potential of the floatingdiffusion FD is transmitted to the output interconnection 2309 via theamplifying transistor 2304 and the selecting transistor 2305. Thereby,pixel signals corresponding to the reset potential and the signalpotential are output from the unit pixel to the output interconnection2309.

In the example illustrated in FIG. 37, the reset interconnection 2306and the TX interconnection 2307 are common to the 16 pixels that formthe pixel block 2131. That is, the reset pulses and the transfer pulsesare, respectively, applied simultaneously to all the 16 pixels.Accordingly, all the pixels that form the pixel block 2131 startelectrical charge accumulation at the same timing, and end electricalcharge accumulation at the same timing. Note that however pixel signalsthat correspond to accumulated electrical charges are output selectivelyto the output interconnection 2309 upon sequential application ofselection pulses to the respective selecting transistors 2305. Also, thereset interconnection 2306, the TX interconnection 2307, and the outputinterconnection 2309 are provided separately for each pixel block 2131.

By configuring a circuit on the basis of the pixel block 2131 in thismanner, an electrical charge accumulation period can be controlled foreach pixel block 2131. In other words, adjacent pixel blocks 2131 can becaused to output pixel signals for different electrical chargeaccumulation periods. Furthermore, by causing one pixel block 2131 torepeat electrical charge accumulation several times and output a pixelsignal at each time while another pixel block 2131 is caused to performelectrical charge accumulation once, these pixel blocks 2131 can becaused to output respective frames for a motion image at different framerates. Note that at least a part of each transistor and eachinterconnection that is illustrated in FIG. 37 functions as a readoutcircuit that reads out pixel signals output from each pixel. The readoutcircuit is provided to each pixel. A part of a configuration of thereadout circuit for each pixel, such as interconnections, may be sharedamong pixels.

FIG. 38 illustrates a part of a configuration of the imaging element2100, and its operation examples. The imaging element 2100 in thepresent example further has a storage unit 2114 in addition to theconfiguration illustrated in FIG. 35. Note that the storage unit 2114may be provided to the signal processing chip 2111. In this case, theimaging element 2100 does not have to have the memory chip 2112. Also,the storage unit 2114 may be provided to the memory chip 2112.

The imaging chip 2113 has an imaging area 2700 in which a plurality ofpixels that respectively generate pixel signals according to incidentlight are placed. For the sake of convenience of explanation, in FIG.38, three (in the row direction) (three (in the column direction) pixelblocks 2131 are illustrated. The numbers of pixels included in eachpixel block 2131 are preferably the same. Also, the number of pixelsincluded in each pixel block 2131 within the imaging area 2700 is fixed.

The signal processing chip 2111 in the present example has, for eachpixel block 2131, a multiplexer 2411, an A/D converter 2412, ade-multiplexer 2413, a control unit 2740, and an arithmetic circuit2415. The multiplexer 2411 sequentially selects pixels included in thecorresponding pixel block 2131, and inputs pixel signal corresponding tothe selected pixels to the A/D converter 2412. The A/D converter 2412converts analog pixel signals into digital pixel data, and inputs it tothe de-multiplexer 2413. The de-multiplexer 2413 causes a storage areacorresponding to the pixel to store the pixel data in a correspondingstorage block 2730. The respective storage blocks 2730 pass the storedpixel data over to the arithmetic circuit 2415 in the subsequent step.

The storage unit 2114 is provided corresponding to a plurality of pixelblocks 2131, and has a plurality of the storage blocks 2730 that canstore pixel data of respectively corresponding pixel blocks 2131. Thestorage block 2730 corresponds one-to-one to the pixel block 2131. Thestorage block 2730 may be connected with the corresponding pixel block2131 via a bus 2720. The storage block 2730 may be a buffer memory.

Also, at least a part of the storage block 2730 can store pixel data ofa pixel block other than the corresponding pixel block 2131. That is, asingle storage block 2730 may be shared by a plurality of the pixelblocks 2131. In other words, the control unit 2740 can cause pixel dataof a single pixel block 2131 to be stored in a plurality of the storageblocks 2730. Because a plurality of the storage blocks 2730 can beutilized efficiently by sharing the storage blocks 2730 as describedbelow, the memory capacity of the entire storage unit 2114 can besuppressed.

Note that about all the pixel blocks 2131, preferably pixel data can bewritten in and read from at least one other storage block 2730 otherthan the corresponding storage block 2730. The other storage block 2730may be predetermined for each pixel block 2131, or may be dynamicallychangeable. Also about all the storage blocks 2730, preferably pixeldata is written in and read from at least one other pixel block 2131other than the corresponding pixel block 2131. The other pixel block2131 may be predetermined for each storage block 2730, or may bedynamically changeable.

Note that each storage block 2730 may be a memory that is provided toeach pixel block 2131 in an area of the signal processing chip 2111 thatoverlaps with a corresponding pixel block 2131. That is, the storageblock 2730 may be provided in an area immediately below a correspondingpixel block 2131 in the signal processing chip 2111. In this case, thepixel block 2131 and the storage block 2730 may be electricallyconnected via TSV. Also, the corresponding storage block 2730, A/Dconverter 2412, arithmetic circuit 2415 and the like are provided in theabove-described area of the signal processing chip 2111 that overlapswith each pixel block 2131. Also, each storage block 2730 may be amemory that is provided outside an area of the signal processing chip2111 that overlaps with the imaging area 2700.

Also, when the respective storage block 2730, A/D converter 2412, andarithmetic circuit 2415 are provided in an area that overlaps with acorresponding pixel block 2131, and when the respective storage block2730 stores pixel data of a pixel block 2131 other than thecorresponding pixel block 2131, an analog pixel signal or a digitalpixel data may be transmitted to an area where the storage block 2730 isprovided. In the former case, the A/D converter 2412 that corresponds tothe storage block 2730 converts the pixel signal into pixel data, andinputs it to the storage block 2730. In the latter case, the pixelsignal is converted into pixel data in the A/D converter 2412 in thearea that overlaps with the pixel block 2131, and then the pixel data istransmitted to a storage block 2730 where the pixel data should bestored. Interconnections for transmitting these pixel signals or pixeldata are provided in the signal processing chip 2111.

The arithmetic circuit 2415 described below processes pixel data storedin the storage block 2730, and passes it over to the image processingunit 2511 in the subsequent step. The arithmetic circuit 2415 may beprovided in the signal processing chip 2111. Note that although, in thefigure, connections for a single pixel block 2131 are illustrated,connections actually exist for each pixel block 2131, and operate inparallel. The arithmetic circuit 2415 is preferably provided to eachpixel block 2131.

As described above, the output interconnection 2309 is providedcorresponding to each of the pixel blocks 2131. Because the imagingelement 2100 is formed by layering the imaging chip 2113, the signalprocessing chip 2111, and the storage unit 2114, the outputinterconnection 2309 can be routed without increasing the size of eachchip in the plane direction by using inter-chip electrical connectionsthat use bumps 2109 for the interconnection.

Note that rate information about a frame rate of each pixel block 2131is provided to the control unit 2740. The control unit 2740 selects astorage block 2730 that should store pixel data of a high frame ratepixel block 2131 based on the rate information. For example, the controlunit 2740 selects a storage block 2730 that corresponds to a referenceframe rate pixel block 2131 as a storage block 2730 that should storethe pixel data.

Note that, in the example illustrated in the figures, the arithmeticcircuit 2415 is provided to each pixel block 2131 including a pluralityof pixels. However, the arithmetic circuit 2415 may be provided to asingle pixel. Note that the arithmetic circuit 2415 may not be providedall the pixels. In other words, at least a first pixel and a secondpixel are disposed in the imaging area 2700, and the imaging element2100 has at least a first arithmetic circuit 2415 that corresponds tothe first pixel and a second arithmetic circuit 2415 that corresponds tothe second pixel.

A first pixel signal output by the first pixel is read out by a firstreadout circuit, and a second pixel signal output by the second pixel isread out by a second readout circuit. The first arithmetic circuit 2415computes a first evaluation value based on the first pixel signal outputby the first pixel, and transmits it to the image processing unit 2511in the subsequent step. The second arithmetic circuit 2415 computes asecond evaluation value based on the second pixel signal output by thesecond pixel, and transmits it to the image processing unit 2511 in thesubsequent step. Here, an evaluation value is obtained by performing apredetermined computation by using a value of a pixel signal. Forexample, the evaluation value may be, a difference between or an averageof a value of a pixel signal output by a predetermined pixel and a valueof an adjacent pixel signal output by an adjacent pixel adjacent to theabove-mentioned pixel. Also, the evaluation value may be, for example, adifference between or an average of values of a plurality of pixelsignals output by a predetermined pixel in different frames. Variousparameters may be used for the computation.

FIG. 39 is a block diagram illustrating a configuration of an imagingdevice according to the present embodiment. An imaging device 2500includes an imaging lens 2520 as an imaging optical system, and theimaging lens 2520 guides a subject luminous flux that is incident alongan optical axis OA to the imaging element 2100. The imaging lens 2520may be a replaceable lens that can be attached/detached to and from theimaging device 2500. The imaging device 2500 includes, mainly, theimaging element 2100, a system control unit 2501, a drive unit 2502, aphotometry unit 2503, a work memory 2504, a recording unit 2505, and adisplay unit 2506.

The imaging lens 2520 is configured with a plurality of optical lensgroups, and forms an image of a subject luminous flux from a scene nearits focal plane. Note that, in FIG. 35, the imaging lens 2520 isrepresentatively shown with a single virtual lens that is placed nearthe pupil. The drive unit 2502 is a control circuit that executeselectrical charge accumulation control such as timing control and areacontrol on the imaging element 2100 according to instructions from thesystem control unit 2501. In this sense, it can be said that the driveunit 2502 serves functions of an imaging element control unit thatcauses the imaging element 2100 to execute electrical chargeaccumulation and output pixel signals.

The imaging element 2100 passes pixel signals over to an imageprocessing unit 2511 of the system control unit 2501. The imageprocessing unit 2511 performs various types of image processing by usingthe work memory 2504 as a workspace, and generates image data. The imageprocessing unit 2511 in the subsequent step of the first and secondarithmetic circuits 2415 performs image processing on first pixel dataof an image that corresponds to the first pixel signal based on thefirst evaluation value received from the first arithmetic circuit 2415,and performs image processing on second pixel data of an image thatcorresponds to the second pixel signal based on the second evaluationvalue received from the second arithmetic circuit 2415. For example,when image data in a JPEG file format is generated, compressionprocesses are executed after color video signals are generated fromsignals obtained from Bayer arrays. The generated image data is recordedin the recording unit 2505 and converted into display signals, and isdisplayed on the display unit 2506 for a preset period of time. Notethat the image processing unit 2511 may be provided in the imagingelement 2100, or may be provided in the system control unit 2501external to the imaging element 2100. Also, the image processing unit2511 may be provided to each pixel, or may be provided to each pixelblock 2131 including a plurality of pixels.

The photometry unit 2503 detects luminance distribution of a scene priorto an imaging sequence for generating image data. The photometry unit2503 includes an AE sensor of approximately one million pixels, forexample. A computing unit 2512 of the system control unit 2501calculates luminance of respective areas within a scene, upon receivingan output of the photometry unit 2503. The computing unit 2512 decides ashutter speed, a diaphragm value, and an ISO speed according to thecalculated luminance distribution. The imaging element 2100 may doubleas the photometry unit 2503. Note that the computing unit 2512 executesvarious types of computation for operating the imaging device 2500.

The drive unit 2502 may be partially or entirely mounted on the imagingchip 2113, or partially or entirely mounted on the signal processingchip 2111. The system control unit 2501 may be partially mounted on theimaging chip 2113 or the signal processing chip 2111. Note that, in theimaging device 2500 in the present example, at least a part of the imageprocessing functions of the image processing unit 2511 are provided tothe imaging element 2100.

FIG. 40 is a functional block diagram of the image processing unit. Theimage processing unit 2511 in the present example extracts pixel blocks2131 that operate at a reference frame rate (a peripheral area 2176described below) and pixel blocks 2131 that operate at a high frame rate(an attention area 2172 described below). The image processing unit 2511has, in addition to the above-described functions, a subject estimatingunit 2150, a group selecting unit 2152, a motion image generating unit2154, and a motion image synthesizing unit 2156. Each of these functionsis described below.

FIG. 41 is a flowchart that illustrates operations of an imaging deviceto generate and record a motion image. FIGS. 42 and 43 each illustrateone example of an image imaged by an imaging element. FIG. 44illustrates a relationship between respective frame rates and outputtiming of image signals.

Operations in FIG. 41 start when a user instructs the imaging device2500 to generate a motion image for example by pressing down a recordbutton. First, the subject estimating unit 2150 drives the drive unit2502 to acquire image data based on image signals from the imagingelement 2100, and estimate a main subject included in an image indicatedby the image data (S2100).

In this case, the drive unit 2502 preferably causes image signals frompixel blocks 2131 included in an entire imaging area, for example allthe pixel blocks 2131, to be output. Also, the drive unit 2502 may causeimage signals from all the pixels included in each pixel block 2131 tobe output, or causes image signals from pixels that are thinned at apredetermined thinning rate to be output. The subject estimating unit2150 compares a plurality of images obtained from the imaging element2100 in a time-series, and identifies a moving subject as a mainsubject. Note that another method may be used to estimate a mainsubject.

For example, when the subject estimating unit 2150 acquires an image2170 in FIG. 42 and an image 2178 in FIG. 43 from the imaging element2100 as temporally sequential images, based on differences therebetween,the subject estimating unit 2150 identifies a child as a main subject2171. Note that grid lines in the image 2170 and the image 2178 indicateboundaries of the pixel blocks 2131, but the number of the pixel blocks2131 is merely an example, and is not limited to the number shown in thefigures.

The group selecting unit 2152 selects at least one pixel block 2131 onwhich image light of the main subject 2171 estimated by the subjectestimating unit 2150 is incident (S2102). For example, pixel blocks 2131including at least a part of the main subject 2171 are selected in theimage 2170. Furthermore, considering that the main subject 2171 moves inan imaging area, the group selecting unit 2152 preferably selects pixelblocks 2131 that further surround the pixel blocks 2131 including atleast a part of the main subject 2171.

The group selecting unit 2152 handles a set of these selected pixelblocks 2131 as an attention area 2172. Furthermore, the group selectingunit 2152 handles, as a peripheral area 2176, a set of pixel blocks 2131not included in the attention area 2172 in the entire imaging area. Thegroup selecting unit 2152 identifies area information 2174 thatindicates a range of the attention area 2172 in relation to the entireimaging area.

In the example illustrated in FIG. 42, the attention area 2172 is arectangular area including total 28 pixel blocks 2131 (seven in thehorizontal direction (four in the vertical direction). On the otherhand, the peripheral area 2176 includes 98 pixel blocks 2131 excludingthe attention area 2172 from total 126 pixel blocks 2131 (21 in thehorizontal direction (six in the vertical direction) which constitutethe imaging area. Also, the position (9, 2) of the attention area 2172in the imaging area that is counted from the left side and the upperside of the upper left end pixel block 2131 in the figure is identifiedas the area information 2174. Furthermore, the numbers in the horizontaland vertical directions, 7 (4, of the attention area 2172 are identifiedas size information.

The group selecting unit 2152 transmits information for identifying thepixel blocks 2131 included in the attention area 2172, and informationfor identifying the peripheral area 2176 to the drive unit 2502. In thiscase, information on frame rates to be applied to the attention area2172 and the peripheral area 2176, respectively, is transmittedtogether. Here, the frame rate to be applied to the attention area 2172is preferably higher than the frame rate to be applied to the peripheralarea 2176. For example, when the frame rate to be applied to theperipheral area 2176 is 60 fps, the frame rate to be applied to theattention area 2172 is set to 180 fps. Preferably, values of the framerates are preset, and stored such that the group selecting unit 2152 canrefer to them, but may be changeable with an operation of a userafterwards.

The drive unit 2502 drives the imaging element 2100 to perform imagingat the respective frame rates (S2104). That is, the drive unit 2502causes the pixel blocks 2131 included in the attention area 2172 toexecute electrical charge accumulation and image signal output at a highframe rate, and causes the pixel blocks 2131 included in the peripheralarea 2176 to execute electrical charge accumulation and image signaloutput at a low frame rate. In other words, the drive unit 2502 obtainsimage signals that correspond to a plurality of frames that arecontiguous in a time-series for the pixel blocks 2131 included in theattention area 2172 while obtaining image signals that correspond to asingle frame for the pixel blocks 2131 included in the peripheral area2176.

For example, when the frame rate of the peripheral area 2176 is set to60 fps and the frame rate of the attention area 2172 is set to 180 fps,as illustrated in FIG. 44, the drive unit 2502 obtains image signals ofthree frames A1, A2, A3 from the attention area 2172 during time 1/60 sin which image signals of a single frame B1 are obtained from theperipheral area 2176 ( 1/60 s=3× 1/180 s). In this case, the drive unit2502 obtains image signals at different frame rates by separatelydriving a set of the reset transistors 2303, the transfer transistors2302, and the selecting transistors 2305 of the pixel blocks 2131included in the peripheral area 2176, and a set of the reset transistors2303, the transfer transistors 2302, and the selecting transistors 2305of the pixel blocks 2131 included in the attention area 2172.

Note that FIG. 44 illustrates timing of outputting image signals, butdoes not illustrate length of an exposure period. The drive unit 2502drives the above-described sets of the transistors for the peripheralarea 2176 and for the attention area 2172 such that the exposure periodpreviously calculated by the computing unit 2512 can be attained.

In addition to this, the length of the exposure period may be changedaccording to frame rates. For example, in the example illustrated inFIG. 44, the exposure period of one frame of the peripheral area 2176may be set to ⅓, which is substantially the same with that for theattention area 2172. Also, image signals may be corrected by the ratioof the frame rates after outputting the image signals. Also, the timingof outputting image signals may not be synchronous as in FIG. 44, butmay be asynchronous between the peripheral area 2176 and the attentionarea 2172.

The image processing unit 2511 sequentially stores, on a frame-by-framebasis, image signals from the attention area 2172 in a predeterminedstorage area of the work memory 2504 (S2106). Similarly, the imageprocessing unit 2511 sequentially stores, on a frame-by-frame basis,image signals from the peripheral area 2176 in a predetermined storagearea of the work memory 2504 (the same step). The work memory 2504 has aplurality of storage blocks 2730 as explained in FIG. 38. The workmemory 2504 may be a memory that includes a memory group thatcorresponds to each pixel block 2131.

The motion image generating unit 2154 reads out the image signals of theattention area 2172 stored in the work memory 2504 (S2108), andgenerates data of the attention area motion image which includes aplurality of frames of the attention area 2172 (S2110). Similarly, themotion image generating unit 2154 reads out the image signals of theperipheral area 2176 stored in the work memory 2504, and generates dataof the peripheral area motion image which includes a plurality of framesof the peripheral area 2176 (the same step). Here, the attention areamotion image and the peripheral area motion image may each be generatedin general-purpose formats such as MPEG and be able to be reproducedseparately, or may each be generated in dedicated formats that do notallow reproduction without going through synthesis processing describedbelow.

FIG. 45 schematically illustrates an attention area motion image and aperipheral area motion image generated by the motion image generatingunit. The motion image generating unit 2154 generates the attention areamotion image at a frame rate that corresponds to a frame rate at whichthe drive unit 2502 drove the attention area 2172. In the exampleillustrated in FIG. 45, the attention area motion image is generated atthe frame rate 1/180 fps which is the same with the frame rate 1/180 fpsat which the drive unit 2502 drove the attention area 2172.

Similarly, the motion image generating unit 2154 generates theperipheral area motion image at a frame rate that corresponds to a framerate at which the drive unit 2502 drove the peripheral area 2176. In theexample illustrated in FIG. 45, the peripheral area motion image isgenerated at the frame rate 1/60 fps which is the same with the framerate 1/60 fps at which the drive unit 2502 drove the peripheral area2176. Note that effective values do not exist in an area of theperipheral area motion image that corresponds to the attention area2172, and the area is indicated with diagonal lines in the figure.

Furthermore, the motion image generating unit 2154 adds headerinformation to the attention area motion image and the peripheral areamotion image, and records the data in the recording unit 2505 (S2112).The header information includes the area information that indicates theposition of the attention area 2172 in relation to the entire imagingarea, the size information that indicates the size of the attention area2172, and timing information that indicates a relationship betweenoutput timing of image signals of the attention area 2172 and outputtiming of image signals of the peripheral area 2176.

The system control unit 2501 determines whether to perform imaging for anext unit time (S2114). Whether to perform imaging of a next unit timeis determined based on whether, at the time point, a user is pressingdown a motion image record button. When imaging is to be performed for anext unit time (S2114: Yes), the flow returns to the above-describedStep S2102, and when imaging is not to be performed for the next unittime (S2114: No), the operation ends.

Here, the “unit time” is preset in the system control unit 2501, andlasts for several seconds. The storage capacity used for storage at StepS2106 is determined based on this unit time, the frame rate and numberof pixel blocks of the attention area 2172, and the frame rate andnumber of pixel blocks of the peripheral area 2176. Based also on thesepieces of information, an area of the storage capacity that stores dataof the attention area 2172 and an area of the storage capacity thatstores data of the peripheral area 2176 are determined.

In this manner, image signals can be obtained at a high frame rate fromthe attention area 2172 including the main subject 2171, and also a dataamount can be reduced by keeping the frame rate for the peripheral area2176 low. Accordingly, as compared with high speed readout from all thepixels, loads of driving and image processing can be reduced, and powerconsumption and heat generation can be suppressed.

Note that when a next unit time starts in the example illustrated inFIG. 41, pixel blocks 2131 are selected again at Step S2102, and thearea information and the size information are updated. Thereby, theattention area 2172 can be updated successively by tracking the mainsubject 2171. In the example illustrated in FIG. 45, in a first frame A7of the unit time in the attention area motion image, an attention area2182 including pixel blocks 2131 that are different from those of a lastframe A6 in the previous unit time are selected, and in accordance withthis, area information 2184 and a peripheral area 2186 are updated.

FIG. 46 illustrates one example of the header information added by themotion image generating unit. The header information in FIG. 46 includesattention area motion image IDs that identify attention area motionimages, frame rates of the attention area motion images, peripheral areamotion image IDs that identify peripheral area motion imagescorresponding to the attention area motion images, frame rates of theperipheral area motion images, timing information, area information, andsize information. These pieces of the header information may be added asthe header information to either one or both of the attention areamotion image and the peripheral area motion image.

FIG. 47 is a flowchart that illustrates operations of an imaging deviceto reproduce and display a motion image. The operations start when auser specifies any of attention area motion images displayed asthumbnails on the display unit 2506, and presses down a reproductionbutton.

The motion image synthesizing unit 2156 reads out, from the recordingunit 2505, data of an attention area motion image specified by the user(S2150). The motion image synthesizing unit 2156 reads out, from therecording unit 2505, data of a peripheral area motion imagecorresponding to the attention area motion image (S2152).

In this case, the motion image synthesizing unit 2156 identifies theperipheral area motion image based on a peripheral area motion image IDindicated in the header information of the attention area motion imageread out at Step S2150. Instead of this, a peripheral area image thatincludes, as the header information, timing information which is thesame with the timing information indicated in the header information ofthe attention area motion image may be searched for and identified.

Note that the header information is included in the attention areamotion image in the above-described example. On the other hand, when theheader information is not included in the attention area motion image,but in the peripheral area motion image, the user may be, previously atStep S2150, caused to specify the peripheral area motion image which isto be read out, and the attention area motion image is specified andread out from the header information at Step S2152.

The motion image synthesizing unit 2156 synthesizes a frame of theattention area motion image and a frame of the peripheral area motionimage into a frame of a displayed motion image (S2154). In this casefirst, the first frame A1 of the attention area motion image is fittedat a position indicated by the area information 2174 in the first frameB1 of the peripheral area motion image to form a synthesized first frameC1 of the displayed motion image. As illustrated in FIG. 45, the motionimage synthesizing unit 2156 causes the first frame C1 of the displayedmotion image to be displayed on the display unit 2506 (S2156).

The motion image synthesizing unit 2156 determines whether there is anext frame of the attention area motion image before a next frame B2 ofthe peripheral area motion image (S2158). When there is a next frame ofthe attention area motion image (S2158: Yes), the motion imagesynthesizing unit 2156 updates the attention area 2172 by using the nextframes A2, A3, and keeps the peripheral area 2176 at the previous frameB1 (S2162) to form next synthesized frames C2, C3 of the displayedmotion image (S2162), and display them sequentially (S2156).

On the other hand, when there is not a next frame of the attention areamotion image before the next frame B2 of the peripheral area motionimage at Step S2158 (S2158), the motion image synthesizing unit 2156updates the attention area 2172 by using a next frame A4 and updatesalso the peripheral area 2176 by using the next frame B2 (S2164) to forma next synthesized frame C4 of the displayed motion image (S2162), anddisplay it (S2156).

As long as there is a next frame of the peripheral area 2176 in theperipheral area motion image (S2160: Yes), Steps S2154 to S2160 arerepeated. When there is not a next frame of the peripheral area 2176 inthe peripheral area motion image (S2160: No), the motion imagesynthesizing unit 2156 makes a search to determine whether, at a unittime next to the unit time of the set of the attention area motion imageand the peripheral area motion image, there is a set of an attentionarea motion image and a peripheral area motion image (S2166). Forexample, the motion image synthesizing unit 2156 makes a search in thesame folder of the recording unit 2505 to determine whether there isanother attention area motion image whose header information includestiming information indicating timing that is immediately after timingindicated by timing information of the previous attention area motionimage.

As long as there is a set of an attention area motion image and aperipheral area motion image in a next unit time (S2166: Yes), StepsS2150 to S2166 are repeated. When there is not a set of an attentionarea motion image and a peripheral area motion image in a next unit time(S2166: No), the operation ends.

In this manner, a smooth motion image can be displayed about theattention area 2172 in which the main subject 2171 is included whilereducing the overall data amount. Note that although at Step S2162, theattention area 2172 is updated directly by using the next frames to formthe synthesized frames of the displayed image, the method of synthesisis not limited thereto. As another example, the boundary line of themain subject 2171 in the attention area 2172 may be identified by imageprocessing, the main subject 2171 surrounded by the boundary line may beupdated with a next frame, and the outside of the boundary line of themain subject 2171 may be kept at the previous frame even if it is withinthe attention area 2172, to form a synthesized frame with the peripheralarea 2176. That is, the frame rate of the outside of the boundary linein the attention area 2172 may be lowered to the frame rate of theperipheral area 2176. Thereby, it is possible to prevent boundaries ofsmoothness in the displayed motion image from looking unnatural. Also,the frame rates of reproduction need not be the same with the framerates at the time of imaging (180 fps for the attention area, and 60 fpsfor the peripheral area), but the frame rates may be for example 60 fpsand 20 fps for the attention area and the peripheral area, respectively.In such a case, the reproduction is slow-motion reproduction.

FIG. 48 is a flowchart that illustrates another example of operations ofthe imaging device to generate and record a motion image. Operations ofFIG. 48 that are the same with those of FIG. 41 are given the samereference numbers, and explanation thereof is omitted.

In the operations of FIG. 48, in addition to or instead of the framerates in FIG. 41, thinning rates are made different between theattention area 2172 and the peripheral area 2176. More specifically, atStep S2120, the drive unit 2502 causes the pixel blocks 2131 included inthe attention area 2172 to execute electrical charge accumulation andimage signal output of pixels that are thinned at a low thinning rate,and causes the pixel blocks 2131 included in the peripheral area 2176 toexecute electrical charge accumulation and image signal output of pixelsthat are thinned at a high thinning rate. For example, pixels in thepixel blocks 2131 included in the attention area 2172 that are thinnedat the thinning rate of 0, that is, all the pixels are read out, andpixels in the pixel blocks 2131 included in the peripheral area 2176that are thinned at the thinning rate of 0.5, that is, a half of thepixels are read out.

In this case, the drive unit 2502 obtains image signals at differentthinning rates by separately driving a set of the reset transistors2303, the transfer transistors 2302, and the selecting transistors 2305of the pixel blocks 2131 included in the peripheral area 2176, and a setof the reset transistors 2303, the transfer transistors 2302, and theselecting transistors 2305 of the pixel blocks 2131 included in theattention area 2172.

At Step S2110, the motion image generating unit 2154 generates anattention area motion image that corresponds to the attention area 2172based on image signals of the attention area 2172 output at a lowthinning rate. The motion image generating unit 2154 similarly generatesa peripheral area motion image that corresponds to the peripheral area2176 based on the image signals of the peripheral area 2176 output at ahigh thinning rate. Also at Step S2112, the motion image generating unit2154 records the attention area motion image and the peripheral areamotion image, with information on the respective thinning rates beingadded thereto, in the recording unit 2505.

FIG. 49 illustrates an example of pixels 2188 to be read out at thethinning rate of 0.5 in one pixel block. In the example illustrated inFIG. 49, when a pixel block 2132 in the peripheral area 2176 is a Bayerarray, the pixels 2188 to be read out and pixels not to be read out areset for every other Bayer array, that is, every two pixels alternatelyin the vertical direction. Thereby, thinned readout can be performedwithout losing a color balance.

FIG. 50 is a flowchart that illustrates operations, corresponding toFIG. 48, of the imaging device to reproduce and display a motion image.Operations of FIG. 50 that are the same with those of FIG. 47 are giventhe same reference numbers, and explanation thereof is omitted.

At Step S2170 in FIG. 50, the motion image synthesizing unit 2156complements pixels of a frame of the peripheral area motion image tomatch its resolution with the resolution of a frame of the attentionarea motion image, and thereafter fits the frame of the attention areamotion image to the frame of the peripheral area motion image; thereby,a synthesized frame of the displayed image is formed. Thereby, imagesignals can be obtained at a high resolution from the attention area2172 including the main subject 2171, and also the data amount can bereduced by keeping the resolution of the peripheral area 2176 low.Accordingly, as compared with high speed readout from all the pixels,loads of driving and image processing can be reduced, and powerconsumption and heat generation can be suppressed.

Note that although the attention area 2172 is a rectangle in theexamples illustrated in FIGS. 35 to 50, the shape of the attention area2172 is not limited thereto. The attention area 2172 may be a convex orconcave polygon, or may have a doughnut shape with the peripheral area2176 positioned inside thereof or another shape as long as the attentionarea 2172 conforms to the boundary line of the pixel blocks 2131. Also,a plurality of the attention areas 2172 that are spaced apart from eachother may be set. In such a case, mutually different frame rates may beset for the attention areas 2172.

Also, frame rates of the attention area 2172 and the peripheral area2176 may be variable. For example, the moving amount of the main subject2171 may be detected with the elapse of a unit time, and a higher framerate may be set for the attention area 2172 if the moving amount of themain subject 2171 is larger. Also, selection of pixel blocks 2131 thatshould be included in the attention area 2172 may be updated at any timeduring the unit time, by tracking the main subject 2171.

Although motion image generation in FIGS. 41 and 48 starts when a userpresses down a record button, and motion image reproduction in FIGS. 47and 50 starts when a user presses down a reproduction button, thestarting time points are not limited thereto. As another example,triggered by a single button operation by a user, an operation of motionimage generation and an operation of motion image reproduction may becontinuously executed, and a through-image display (or also called alive view display) may be performed on the display unit 2506. In thiscase, a display for causing the user to recognize the attention area2172 may be superimposed. For example, a frame may be displayed over theboundary of the attention area 2172 on the display unit 2506, or theluminance of the peripheral area 2176 may be lowered or the luminance ofthe attention area 2172 may be raised.

In the operations in FIG. 48, thinning rates are made different betweenthe attention area 2172 and the peripheral area 2176. Instead of makingthe thinning rates different, the numbers of adjacent rows of pixelswhose pixel signals are added may be made different. For example, in theattention area 2172, the number of rows is one, which means that pixelsignals are output without addition among adjacent rows, and in theperipheral area 2176, the number of rows is larger than that for theattention area 2172, that is, for example two, which means that pixelsignals of pixels of two adjacent rows that are in the same columns areoutput. Thereby, similar to FIG. 48, the overall signal amount can bereduced while keeping the resolution of the attention area 2172 higherthan that of the peripheral area 2176.

Note that the motion image synthesizing unit 2156 may be provided in anexternal display apparatus, for example a PC, instead of being providedin the image processing unit 2511 of the imaging device 2500. Also, theabove-described embodiment may be applied not only to motion imagegeneration, but also to still image generation.

Also, although in the above-described embodiments, a plurality of thepixel blocks 2131 is divided into two areas, the attention area 2172 andthe peripheral area 2176, the number of division is not limited thereto,and the pixel blocks 2131 may be divided into three or more areas. Inthis case, pixel blocks 2131 that correspond to the boundary between theattention area 2172 and the peripheral area 2176 may be handled as aboundary area, and the boundary area may be controlled by using anintermediate value between a value of a control parameter used for theattention area 2172 and a value of a control parameter used for theperipheral area 2176. Thereby, it is possible to prevent the boundarybetween the attention area 2172 and the peripheral area 2176 fromlooking unnatural.

Accumulation periods and numbers of times of accumulation of electricalcharges, and the like may be made different between the attention area2172 and the peripheral area 2176. In this case, the attention area 2172and the peripheral area 2176 may be divided based on luminance, andfurthermore an intermediate area may be provided.

FIGS. 51A and 51B are diagrams for explaining an example of a scene andarea division. FIG. 51A illustrates a scene captured by an imaging areaof the imaging chip 2113. Specifically, the scene includessimultaneously a shadowed subject 2601 and an intermediate subject 2602included in an indoor environment, and a highlighted subject 2603 of anoutdoor environment observed within a window frame 2604. When imaging,with a conventional imaging element, such a scene in which the contrastbetween a highlighted portion and a shadowed portion is high, blocked-upshadows occur at the shadowed portion if electrical charge accumulationis executed by using the highlighted portion as a reference, andblown-out highlights occur at the highlighted portion if electricalcharge accumulation is executed by using the shadowed portion as areference. That is, it can be said that, for a high contrast scene, thephoto diode does not have a sufficient dynamic range that is needed forimage signals to be output by one-time electrical charge accumulationthat is uniform for the highlighted portion and the shadowed portion. Tocope with this, in the present embodiment, a scene is divided intopartial areas such as a highlighted portion and a shadowed portion, andsubstantial expansion of a dynamic range is attempted by making thenumbers of times of electrical charge accumulation mutually differentbetween photo diodes that correspond to respective areas.

FIG. 51B illustrates area division of an imaging area in the imagingchip 2113. The computing unit 2512 analyzes the scene of FIG. 51Acaptured by the photometry unit 2503 to divide the imaging area based onluminance. For example, the system control unit 2501 causes thephotometry unit 2503 to execute scene acquisition multiple times whilechanging exposure periods, and the computing unit 2512 decides divisionlines of the imaging area by referring to changes in distribution ofblown-out highlight areas and blocked-up shadowed areas. In the exampleof FIG. 51B, the computing unit 2512 performs division into three areas,a shadowed area 2611, an intermediate area 2612, and a highlighted area2613.

The division line is defined along boundaries of pixel blocks 2131. Thatis, each divided area includes an integer number of groups. Then, pixelsof each group included in the same area perform electrical chargeaccumulation and pixel signal output the same number of times in aperiod that corresponds to a shutter speed decided by the computing unit2512. If pixels belong to different areas, electrical chargeaccumulation and pixel signal output are performed different numbers oftimes.

FIG. 52 is a diagram for explaining electrical charge accumulationcontrol for the respective areas divided in the example in FIGS. 51A and51B. Upon receiving an imaging stand-by instruction from a user, thecomputing unit 2512 decides a shutter speed T₀ based on an output fromthe photometry unit 2503. Furthermore, the computing unit 2512 performsdivision into the shadowed area 2611, the intermediate area 2612, andthe highlighted area 2613 in a manner as above-described, and decidesthe numbers of times of electrical charge accumulation based onrespective pieces of luminance information. The numbers of times ofelectrical charge accumulation are decided such that pixels are notsaturated by one-time electrical charge accumulation. For example, thenumbers of times of electrical charge accumulation are decided such that80 to 90% of accumulatable electrical charges is accumulated in aone-time electrical charge accumulation operation.

Here, electrical charge accumulation is performed once for the shadowedarea 2611. That is, the decided shutter speed T₀ and the electricalcharge accumulation period are caused to match. Also, electrical chargeaccumulation is performed twice for the intermediate area 2612. That is,a one-time electrical charge accumulation period is set to T₀/2, andelectrical charge accumulation is repeated twice during the shutterspeed T₀. Also, electrical charge accumulation is performed four timesfor the highlighted area 2613. That is, a one-time electrical chargeaccumulation period is set to T₀/4, and electrical charge accumulationis repeated four times during the shutter speed T₀.

Upon receiving an imaging instruction from a user at a clock time t=0,the drive unit 2502 applies reset pulses and transfer pulses to pixelsin groups belonging to the respective areas. This application triggers astart of electrical charge accumulation of all the pixels.

At a clock time t=T₀/4, the drive unit 2502 applies transfer pulses topixels in groups belonging to the highlighted area 2613. Then, the driveunit 2502 sequentially applies selection pulses to pixels in each groupto cause their respective pixel signals to be output to the outputinterconnection 2309. After pixel signals of all the pixels in thegroups are output, the drive unit 2502 applies reset pulses and transferpulses again to pixels in groups belonging to the highlighted area 2613to cause second electrical charge accumulation to be started.

Note that because selective output of pixel signals takes time, a timelag occurs between the end of first electrical charge accumulation andthe start of second electrical charge accumulation. When this time lagis substantially negligible, a one-time electrical charge accumulationperiod may be calculated by dividing the shutter speed T₀ by the numbersof times of electrical charge accumulation as described above. On theother hand, if not negligible, the shutter speed T₀ may be adjusted byconsidering the time, or the a one-time electrical charge accumulationperiod may be made shorter than the time obtained by dividing theshutter speed T₀ by the numbers of times of electrical chargeaccumulation.

At a clock time t=T₀/2, the drive unit 2502 applies transfer pulses topixels in groups belonging to the intermediate area 2612 and thehighlighted area 2613. Then, the drive unit 2502 sequentially appliesselection pulses to pixels in each group to cause their respective pixelsignals to be output to the output interconnection 2309. After pixelsignals of all the pixels in the groups are output, the drive unit 2502applies reset pulses and transfer pulses again to pixels in groupsbelonging to the intermediate area 2612 and the highlighted area 2613 tocause second electrical charge accumulation to be started for theintermediate area 2612 and cause third electrical charge accumulation tobe started for the highlighted area 2613.

At a clock time t=3T₀/4, the drive unit 2502 applies transfer pulses topixels in groups belonging to the highlighted area 2613. Then, the driveunit 2502 sequentially applies selection pulses to pixels in each groupto cause their respective pixel signals to be output to the outputinterconnection 2309. After pixel signals of all the pixels in thegroups are output, the drive unit 2502 applies reset pulses and transferpulses again to pixels in groups belonging to the highlighted area 2613to cause fourth electrical charge accumulation to be started.

At the clock time t=T₀, the drive unit 2502 applies transfer pulses topixels of all the areas. Then, the drive unit 2502 sequentially appliesselection pulses to pixels in each group to cause their respective pixelsignals to be output to the output interconnection 2309. According tothe above-described control, pixel signals that correspond to once arestored in each pixel memory 2414 that corresponds to the shadowed area2611, pixel signals that correspond to twice are stored in each pixelmemory 2414 that corresponds to the intermediate area 2612, and pixelsignals that correspond to four times are stored in each pixel memory2414 that corresponds to the highlighted area 2613.

Note that the drive unit 2502 may sequentially apply reset pulses topixels in groups belonging to any area, and sequentially reset pixels inthe groups belonging to the area. Also, the drive unit 2502 maysequentially apply transfer pulses to the reset pixels in the group.Triggered by this application, pixels of each group may sequentiallystart electrical charge accumulation. After the end of electrical chargeaccumulation of pixels in groups belonging to all the areas, the driveunit 2502 may apply transfer pulses to pixels in the all the areas.Then, the drive unit 2502 may sequentially applies selection pulses topixels in each group to cause their respective pixel signals to beoutput to the output interconnection 2309.

These pixel signals are sequentially transferred to the image processingunit 2511. The image processing unit 2511 generates image data with ahigh dynamic range based on the pixel signals. Specific processing isdescribed below.

FIG. 53 is a table that indicates a relationship between the number oftimes of integration and the dynamic range. Pixel data that correspondsto multiple times of repeatedly executed electrical charge accumulationare subjected to an integration process by the image processing unit2511 to form a part of image data with a high dynamic range.

When compared with, as a reference, a dynamic range of an area whosenumber of times of integration is once, that is, for which electricalcharge accumulation is performed once, a dynamic range of an area whosenumber of times of integration is twice, that is, whose output signal isintegrated by performing electrical charge accumulation twice isexpanded by one step. Similarly, when the number of times of integrationis four times, the dynamic range is expanded by two steps, and when thenumber of times of integration is 128, the dynamic range is expanded byseven steps. That is, in order to attempt to obtain n-steps of dynamicrange expansion, output signals may be integrated 2^(n) times.

Here, in order for the image processing unit 2511 to identify how manytimes electrical charge accumulation has been performed for whichdivided area, a 3-bit exponent indicating the number of times ofintegration is added to an image signal. As illustrated, exponents areallocated sequentially, 000 to the number of times of integration once,001 to twice, . . . , 111 to 128 times.

The image processing unit 2511 refers to an exponent of each piece ofpixel data received from the arithmetic circuit 2415 and when a resultof the reference shows that the number of times of integration is two ormore, executes an integration process of the pixel data. For example,when the number of times of integration is two (one step), upper 11 bitsof two pieces of 12-bit pixel data corresponding to electrical chargeaccumulation are added together to generate a single piece of 12-bitpixel data. Similarly, when the number of times of integration is 128(seven steps), upper 5 bits of 128 pieces of 12-bit pixel datacorresponding to electrical charge accumulation are added together togenerate a single piece of 12-bit pixel data. That is, upper bits, thenumber of which is obtained by subtracting, from 12, the number of stepscorresponding to the number of times of integration, are added togetherto generate a single piece of 12-bit pixel data. Note that lower bitsthat are not to be added are eliminated.

By performing processing in this manner, the luminance range thatprovides a gradation can be shifted to the high luminance side inaccordance with the number of times of integration. That is, 12 bits areallocated to a limited range on the high luminance side. Accordingly, agradation can be provided to an image area that conventionally includedblown-out highlights.

Note that however that, because 12 bits are allocated to differentluminance ranges of other divided areas, image data cannot be generatedby synthesis of simply connecting the areas. To cope with this, theimage processing unit 2511 performs a re-quantization process by using,as a reference, a highest luminance pixel and a lowest luminance pixelin order to make all the areas 12-bit image data while preservingobtained gradations as much as possible. Specifically, quantization isexecuted by performing gamma conversion so that the smoother gradationscan be preserved. By performing processing in this manner, image datawith a high dynamic range can be obtained.

Note that the description of the number of times of integration is notlimited to a 3-bit exponent being added to pixel data asabove-described, but the number of times of integration may be describedas accompanying information other than the pixel data. Also, theexponent may be omitted from pixel data, and instead the number of timesof integration may be acquired at the time of an adding process bycounting the number of pieces of pixel data stored in the pixel memory2414.

Also, although in the above-described image processing, are-quantization process to make all the areas 12-bit image data isexecuted, the number of output bits may be increased from the bit numberof pixel data, in accordance with an upper limit number of times ofintegration. For example, if the upper limit number of times ofintegration is defined as 16 (four steps), all the areas may be made,for 12-bit pixel data, 16-bit image data. By performing processing inthis manner, image data can be generated without cancellation of digits.

Next, a series of imaging operation processes is explained. FIG. 54 is aflow diagram showing processing of imaging operations. The flow startswhen a power supply of the imaging device 2500 is turned on.

At Step S2201, the system control unit 2501 waits for a switch SW1 to bepressed down, which is an imaging stand-by instruction. When pressingdown of the switch SW1 is sensed, the flow proceeds to Step S2202.

At Step S2202, the system control unit 2501 executes photometryprocessing. Specifically, upon obtaining an output of the photometryunit 2503, the computing unit 2512 calculates luminance distribution ofa scene. Then, the flow proceeds to Step S2203, and as described above,a shutter speed, area division, the number of times of integration, andthe like are decided.

Upon completion of the imaging stand-by operation, the flow proceeds toStep S2204, and waits for a switch SW2 to be pressed down, which is animaging instruction. At this time, when the elapsed time exceeds apredetermined time Tw (YES at Step S2205), the flow returns to StepS2201. When pressing down of the switch SW2 is sensed before the elapsedtime exceeds the time Tw (NO at Step S2205), the flow proceeds to StepS2206.

At Step S2206, the drive unit 2502 that has received an instruction ofthe system control unit 2501 executes an electrical charge accumulationprocess and a signal readout process that are explained by using FIG.52. Then, upon completion of entire signal readout, the flow proceeds toStep S2207, the image processing explained by using FIG. 53 is executed,and a recording process of recording generated image data in therecording unit is executed.

Upon completion of the recording process, the flow proceeds to StepS2208, and it is determined whether the power supply of the imagingdevice 2500 has been turned off. When the power supply has not beenturned off, the flow returns to Step S2201, and when the power supplyhas been turned off, the series of imaging operation processes ends.

FIG. 55 is a block diagram that illustrates a specific configuration ofthe signal processing chip 2111 as one example. A pixel data processingunit 2910 illustrated in FIG. 55 is provided to each pixel block 2131.Note that however, similar to the arithmetic circuit 2415 explained inrelation to FIG. 38, the pixel data processing unit 2910 may be providedto each of two or more pixels. Also, configurations, among theconfigurations of the pixel data processing unit 2910, other than thearithmetic circuit 2415 may be provided to each pixel block 2131.

The control unit 2740 in the signal processing chip 2111 in the presentexample serves a part of or all the functions of the drive unit 2502.The control unit 2740 includes a sensor control unit 2441, a blockcontrol unit 2442, a synchronization control unit 2443, and a signalcontrol unit 2444 that serve divided control functions, and a drivecontrol unit 2420 that performs overall control on the respectivecontrol units. The drive control unit 2420 converts instructions fromthe system control unit 2501 into control signals that can be executedby the respective control units, and passes them over to the respectivecontrol units.

The sensor control unit 2441 performs transmission control on controlpulses that are to be transmitted to the imaging chip 2113 and relate toelectrical charge accumulation and electrical charge readout of eachpixel. Specifically, the sensor control unit 2441 controls the start andend of electrical charge accumulation by transmitting reset pulses andtransfer pulses to target pixels, and causes pixel signals to be outputto the output interconnection 2309 by transmitting selection pulses toreadout pixels.

The block control unit 2442 executes transmission of specifying pulsesthat are to be transmitted to the imaging chip 2113 and specify a pixelblock 2131 to be controlled. As explained by using FIG. 51B, etc., areasdivided into the attention area 2172 and the peripheral area 2176 mayeach include a plurality of mutually adjacent pixel blocks 2131. Pixelblocks 2131 belonging to the same area form a single block group. Pixelsthat are included in the same block group start electrical chargeaccumulation at the same timing, and end the electrical chargeaccumulation at the same timing. To cope with this, the block controlunit 2442 plays a role of forming blocks of pixel blocks 2131 bytransmitting specifying pulses to pixel blocks 2131 to be targets basedon designation by the drive control unit 2420. Transfer pulses and resetpulses that each pixel receives via the TX interconnection 2307 and thereset interconnection 2306 are logical AND of each pulse transmitted bythe sensor control unit 2441 and specifying pulses transmitted by theblock control unit 2442.

In this manner, by controlling each area as a mutually independent blockgroup, the electrical charge accumulation control explained by usingFIG. 52 can be realized. The drive control unit 2420 may apply resetpulses and transfer pulses to pixels included in the same block group atdifferent timing. Also, after terminating electrical charge accumulationof pixels included in the same block group at the same timing, the drivecontrol unit 2420 may sequentially apply selection pulses to the pixelsin the block group, and sequentially read out their respective pixelsignals.

The synchronization control unit 2443 transmits a synchronization signalto the imaging chip 2113. Each pulse becomes active in the imaging chip2113 in synchronization with the synchronization signal. For example, byadjusting the synchronization signal, random control, thinning control,and the like only on particular pixels among pixels belonging to thesame pixel block 2131 can be realized.

The signal control unit 2444 mainly performs timing control on the A/Dconverter 2412. Pixel signals output via the output interconnection 2309are input to the A/D converter 2412 through a CDS circuit 2410 and themultiplexer 2411. The A/D converter 2412 is controlled by the signalcontrol unit 2444 to convert the input pixel signals into digital pixeldata. The pixel data converted into digital signals is passed over tothe de-multiplexer 2413, and is stored as a pixel value of digital datain the pixel memory 2414 corresponding to each pixel. The pixel memory2414 is one example of the storage block 2730.

The signal processing chip 2111 has a timing memory 2430, as anaccumulation control memory, that stores block division informationabout which pixel blocks 2131 are to be combined to form a block groupof the attention area 2172 and the peripheral area 2176, and informationon the number of times of accumulation about how many times each blockgroup formed repeats electrical charge accumulation. The timing memory2430 is configured for example with a flash RAM.

As described above, which pixel blocks 2131 are to be combined to form ablock group is decided by the system control unit 2501 based on adetection result of luminance distribution detection of a scene that isexecuted prior to a series of imaging sequence. The decided block groupsare divided for example into a first block group, a second block group,. . . , and defined by which pixel blocks 2131 are included therein. Thedrive control unit 2420 receives the block division information from thesystem control unit 2501, and stores it in the timing memory 2430.

Also, the system control unit 2501 decides how many times each blockgroup repeats electrical charge accumulation based on a detection resultof luminance distribution. The drive control unit 2420 receives theinformation on the number of times of accumulation from the systemcontrol unit 2501, and stores it in the timing memory 2430 by pairingthe information on the number of times of accumulation with thecorresponding block division information. By storing the block divisioninformation and the information on the number of times of accumulationin the timing memory 2430 in this manner, the drive control unit 2420may execute a series of electrical charge accumulation controlindependently by successively referring to the timing memory 2430. Thatis, when controlling acquisition of a single image, once the drivecontrol unit 2420 receives a signal of an imaging instruction from thesystem control unit 2501, the drive control unit 2420 thereafter is ableto complete accumulation control without receiving an instruction aboutcontrol on each pixel from the system control unit 2501 each time.

The drive control unit 2420 receives, from the system control unit 2501,block division information and information on the number of times ofaccumulation that are updated based on results of photometry (detectionresults of luminance distribution) executed in synchronization with animaging stand-by instruction, and as appropriate updates stored contentsof the timing memory 2430. For example, the drive control unit 2420updates the timing memory 2430 in synchronization with an imagingstand-by instruction or an imaging instruction. With this configuration,faster electrical charge accumulation control is realized, and thesystem control unit 2501 may execute other processing in parallel withelectrical charge accumulation control executed by the drive controlunit 2420.

The drive control unit 2420 which executes electrical chargeaccumulation control on the imaging chip 2113 further refers to thetiming memory 2430 in execution of readout control. For example, thedrive control unit 2420 refers to information on the number of times ofaccumulation of each block group to store pixel data output from thede-multiplexer 2413 in a corresponding address of the pixel memory 2414.

The drive control unit 2420 reads out target pixel data of each pixelblock from the pixel memory 2414 according to a delivery request fromthe system control unit 2501, and passes it over to the image processingunit 2511. At this time, the drive control unit 2420 passes additionaldata corresponding to the respective pieces of target pixel datatogether over to the image processing unit 2511.

For each of the pixel blocks 2131, the arithmetic circuit 2415 performspredetermined computation on pixel data according to pixel signalsgenerated by the corresponding pixel block 2131. That is, the arithmeticcircuit 2415 is provided corresponding to a pixel block 2131, andperforms computational processing for each of the pixel blocks 2131.Note that the arithmetic circuit 2415 is provided to the pixel block2131 on a one-to-one basis. That is, the arithmetic circuit 2415 isprovided to a signal processing chip 2111 immediately below the pixelblock 2131. The drive control unit 2420 reads out pixel data stored inthe pixel memory 2414, passes it to the arithmetic circuit 2415, andcauses the arithmetic circuit 2415 to execute the predeterminedcomputation processing.

A data transfer interface that transmits pixel data or differential datadescribed below according to a delivery request is provided to the pixelmemory 2414. The data transfer interface is connected with a datatransfer line 2920 that connects with the image processing unit 2511.The data transfer line 2920 is configured for example with a serial bus.In this case, a delivery request from the system control unit 2501 tothe drive control unit 2420 is executed by addressing that utilizes anaddress bus.

A predetermined computation may be executed after acquiring pixel databy using control parameters that are different between the attentionarea 2172 and the peripheral area 2176, by using the signal processingchip 2111 in FIG. 55. For example, although in FIGS. 41 to 44, a motionimage is generated from images that are acquired at frame rates that aredifferent between the attention area 2172 and the peripheral area 2176,instead of this, an S/N ratio may be improved by performing imageprocessing of averaging images acquired at a high frame rate. In thiscase, the drive control unit 2420 obtains pixel signals that correspondto multiple times, for example four times, from the attention area 2172for example while obtaining pixel signals that corresponds to once fromthe peripheral area 2176, and stores the pixel data in the pixel memory2414. The arithmetic circuit 2415 reads out a plurality of pieces ofpixel data obtained, from the pixel memory 2414, for each pixel of theattention area 2172, and averages them for respective pixels. Thereby,random noises of each pixel of the attention area 2172 are reduced, andan S/N ratio of the attention area 2172 can be improved.

Note that a memory 2940 is connected with the data transfer line 2920.The memory 2940 may be a volatile memory that sequentially stores pixeldata from the pixel memory 2414 at designated addresses. For example,the memory 2940 is a DRAM. The memory 2940 stores RGB data thatcorresponds to one frame that uses received pixel data of each pixelblock 2131.

The control unit 2740 causes an arithmetic circuit 2415 corresponding tothe pixel block 2131 to exchange data with arithmetic circuits 2415 thatcorrespond to peripheral pixel blocks 2131. In the example of FIG. 55,the drive control unit 2420 causes data to be transmitted among aplurality of the arithmetic circuits 2415. Each arithmetic circuit 2415receives at least a part of other computation results in otherarithmetic circuits 2415 that correspond to other pixel blocks 2131.Each arithmetic circuit 2415 may generate its own computation resultsbased further on other received computation results.

Also, the arithmetic circuit 2415 inputs, to the output circuit 2922,computation results for the respective pixel blocks 2131 for whichcomputational processing has been performed. The output circuit 2922associates computation results in the arithmetic circuit 2415 with pixeldata, and outputs them to the system control unit 2501. Here, toassociate with the pixel data and output them means that computationresults obtained from processing by the arithmetic circuit 2415 on pixeldata of the pixel block 2131 and information about to which pixel blockthe pixel data subjected to the computation relates are associated witheach other and output.

Note that although data transferred to the system control unit 2501 viathe output circuit 2922 is computation results for each pixel block2131, the system control unit 2501 cannot utilize the received datawithout knowing what types of computation have been performed in eachpixel block 2131 to obtain the received data. In the present example,the output circuit 2922 adds a data code indicating computationalcontents in each arithmetic circuit 2415 to computation results, andoutput them. The data code may be predetermined for each arithmeticcircuit 2415. Also, when the arithmetic circuit 2415 can perform aplurality of types of computation, the arithmetic circuit 2415preferably notifies the output circuit 2922 of information indicatingwhat types of computation has been performed. That is, the outputcircuit 2922 generates contents of performed computation, computationresults, and control information for each pixel block 2131 as a singledata array, and outputs it. Examples of specific data arrays that theoutput circuit 2922 outputs are described below.

FIG. 56 illustrates a plurality of arithmetic circuits 2415 thatexchange computation results with each other. For example, the firstarithmetic circuit 2415 receives a second evaluation value in the secondarithmetic circuit 2415, or computation results in a process in whichthe second arithmetic circuit 2415 computes the second evaluation value.In this case, the first arithmetic circuit 2415 computes a firstevaluation value based on the second evaluation value or the computationresults. Alternatively, each arithmetic circuit 2415 itself may performcomputation on pixel signals that correspond to another arithmeticcircuit 2415 by reading out the pixel signals from a pixel memory 2414that corresponds to the arithmetic circuit 2415. For example, the firstarithmetic circuit 2415 reads out a second pixel signal that correspondsto the second arithmetic circuit 2415. In this case, the firstarithmetic circuit 2415 computes the first evaluation value based on thesecond pixel signal read out.

In the present example, the pixel blocks 2131 that correspond to anarithmetic circuit 2415-1, an arithmetic circuit 2415-2, and anarithmetic circuit 2415-4 is adjacent to each other in the columndirection, and the pixel blocks 2131 that correspond to the arithmeticcircuit 2415-1, an arithmetic circuit 2415-3, and an arithmetic circuit2415-5 are adjacent to each other in the row direction. Each arithmeticcircuit 2415 receives at least a part of other computation results inother arithmetic circuits 2415 that correspond to pixel blocks 2131adjacent to the pixel block 2131 that corresponds to the arithmeticcircuit 2415. Here, being adjacent means not only being adjacent in therow and column directions. The pixel blocks 2131 may be adjacent in adiagonal direction. In the present example, a case where the pixelblocks 2131 are adjacent in the row direction and in the columndirection is explained.

Adjacent arithmetic circuits 2415 are each connected via an output busthat outputs computation results to an arithmetic circuit 2415 thatcorresponds to an adjacent pixel block 2131, and an input bus thatinputs computation results to an arithmetic circuit 2415 thatcorresponds to an adjacent pixel block 2131. The control unit 2740causes an arithmetic circuit 2415 that corresponds to the pixel block2131 to generate its computation results of the pixel block 2131 basedon computation results from arithmetic circuits 2415 that correspond toother adjacent pixel blocks 2131.

FIG. 57 is a block diagram that illustrates one example of aconfiguration of the arithmetic circuit 2415. Each arithmetic circuit2415 has a corresponding block calculating unit 2912, an averagecalculating unit 2913, an average-average calculating unit 2914, aperipheral block calculating unit 2911, and a pixel-average calculatingunit 2915. The input of the corresponding block calculating unit 2912 isconnected to the output of the pixel memory 2414 that corresponds to thepixel block 2131, and the output of the corresponding block calculatingunit 2912 is connected with the input of the average calculating unit2913, the input of the average-average calculating unit 2914, the inputof the output circuit 2922, and each arithmetic circuit 2415 thatcorresponds to an adjacent pixel block 2131. For example, thecorresponding block calculating unit 2912 outputs an average of pixelvalues of the respective colors in the corresponding pixel block 2131.

The peripheral block calculating unit 2911 has a plurality of inputs,and the respective inputs are connected with the outputs of arithmeticcircuits 2415 that correspond to a plurality of pixel blocks 2131 thatare adjacent to the pixel block 2131. The output of the peripheral blockcalculating unit 2911 is connected with the input of the averagecalculating unit 2913. For example, the peripheral block calculatingunit 2911 may calculate an average of averages of pixel values of therespective colors received from other arithmetic circuits 2415. Also,the peripheral block calculating unit 2911 may output, as it is,averages of pixel values of the respective colors received from otherarithmetic circuits 2415.

The average calculating unit 2913 has two input parts, one of the inputsis connected to the output of the corresponding block calculating unit2912, and the other input is connected to the output of the peripheralblock calculating unit 2911. For example, the average calculating unit2913 outputs an average of pixel values of the respective colors in thecorresponding pixel block 2131 and adjacent pixel blocks 2131 based onan average value output by the corresponding block calculating unit2912, and an average value output by the peripheral block calculatingunit 2911.

The average-average calculating unit 2914 has two inputs, one of theinputs is connected with the output of the average calculating unit2913, and the other input is connected to the output of thecorresponding block calculating unit 2912. The output of theaverage-average calculating unit 2914 is connected with the input of theoutput circuit 2922. For example, the average-average calculating unit2914 calculates a difference between an average of pixel values of therespective colors calculated by the average calculating unit 2913, andan average of pixel values of the respective colors calculated by thecorresponding block calculating unit 2912.

The pixel-average calculating unit 2915 has two inputs, one of theinputs is connected with the output of the average calculating unit2913, and the other input is connected with the output of the pixelmemory 2414 that corresponds to the pixel block 2131. The output of thepixel-average calculating unit 2915 is connected with the input of thepixel memory 2414 that corresponds to the pixel block 2131. For example,the pixel-average calculating unit 2915 outputs a difference betweeneach pixel value in the pixel block 2131, and an average value of acorresponding color among averages of pixel values of the respectivecolors calculated by the average calculating unit 2913.

The control unit 2740 transmits computation results in the correspondingblock calculating unit 2912 to other arithmetic circuit 2415 and outputcircuit 2922. Also, the control unit 2740 transmits computation resultsin the average-average calculating unit 2914 to the output circuit 2922.Furthermore, the control unit 2740 feeds back computation results in thepixel-average calculating unit 2915 to the pixel memory 2414 of thepixel block 2131.

Note that each calculating unit of the arithmetic circuit 2415 can beconfigured with an adding circuit, a subtracting circuit, and a dividingcircuit. In this manner, by simplifying the circuit configuration of thearithmetic circuit 2415, the arithmetic circuit 2415 can be implementedfor each pixel block 2131.

FIG. 58 is a flowchart for explaining one example of operations of thearithmetic circuit 2415. After the arithmetic circuit 2415 starts theoperations, at Step S2300, the control unit 2740 reads out, from thepixel memory 2414 that corresponds to the pixel block 2131, RGB pixeldata of the pixel block 2131 imaged at a frame rate of the pixel block2131, and inputs it to the corresponding block calculating unit 2912. AtStep S2310, the control unit 2740 inputs at least a part of computationresults in the adjacent pixel block 2131 from the adjacent arithmeticcircuit 2415 to the peripheral block calculating units 2911 insynchronization with Step S2300. In the present example, each arithmeticcircuit 2415 calculates an average of respective pixel values of RGBpixels, and the peripheral block calculating units 2911 receives theaverage of the respective pixel values of the RGB pixels calculated byadjacent arithmetic circuit 2415.

At Step S2320, the control unit 2740 causes the corresponding blockcalculating unit 2912 to perform predetermined computation on pixel dataof the pixel block 2131 that corresponds to the control unit 2740. Forexample, the corresponding block calculating unit 2912 calculatesrespective added average values (Ar, Ag, Ab) of the RGB pixels of thepixel block 2131. The added average value is calculate as follows:

Ai=Σ(i pixels within a pixel block)/(the number of i pixels within thepixel block)(i=r,g,b)

At Step S2322, the control unit 2740 causes the corresponding blockcalculating unit 2912 to input the average values (Ar, Ag, Ab) to theinput of the output circuit 2922 and the inputs of the respectivearithmetic circuits 2415 that correspond to four adjacent pixel block2131.

At Step S2340, the control unit 2740 causes the peripheral blockcalculating unit 2911 to calculate averages (Br, Bg, Bb) (as adjacentpixel block averages) in the plurality of adjacent pixel blocks 2131based on the respective added average values of the RGB pixels of theadjacent pixel blocks 2131. For example, the adjacent pixel blockaverage is calculated as follows:

Bi=ΣAi/4(i=r,g,b)(note that the number of adjacent pixel blocks 2131 isassumed to be four.)

At Step S2350, the control unit 2740 causes the average calculating unit2913 to perform predetermined computation on other computation resultsreceived from other arithmetic circuits 2415 and computation result inthe corresponding block calculating unit 2912. For example, the averagecalculating unit 2913 calculates overall averages (Cr, Cg, Cb) of thefour adjacent pixel block average values (Br, Bg, Bb) calculated at StepS2340, and the added average values (Ar, Ag, Ab) of the pixel block 2131calculated at Step S2320. The overall average is calculated as follows:

Ci=(Bi+Ai)/2(i=r,g,b)

At Step S2360, the control unit 2740 causes the average-averagecalculating unit 2914 to calculate difference values (ΔAr, ΔAg, ΔAb)between the added average values (Ar, Ag, Ab) in the block calculated bythe corresponding block calculating unit 2912 at Step S2320, and theoverall average values (Cr, Cg, Cb) calculated by the averagecalculating unit 2913 at Step S2350. The difference value is calculatedas follows:

ΔAi=(Ai−Ci) (i=r, g, b) At Step S2370, the control unit 2740 causes theaverage-average calculating unit 2914 to input the difference values(ΔAr, ΔAg, ΔAb) to the output circuit 2922. Note that the arithmeticcircuit 2415 may not have the average-average calculating unit 2914, butinput computation results in the average calculating unit 2913 to theoutput circuit 2922, in place of computation results in theaverage-average calculating unit 2914.

At Step S2380, the control unit 2740 causes the pixel-averagecalculating unit 2915 to calculate difference values (ΔCr, ΔCg, ΔCb)between RGB pixel data of the pixel block acquired at Step S2310, andthe overall average values (Cr, Cg, Cb) calculated by the averagecalculating unit 2913 at Step S2350. The difference value is calculatedas follows:

ΔCi=(Ci−i pixels in the pixel block)(i=r,g,b)

Thereby, original information of pixel data can be stored by using asmall difference value and an average value. That is, based oncomputation results in the average calculating unit 2913, pixel data ofthe pixel block 2131 corresponding to itself can be compressed.

At Step S2390, the control unit 2740 feeds back (ΔCr, ΔCg, ΔCb) to thepixel memory 2414 of the pixel block 2131. At Step S2392, the controlunit 2740 determines whether to continue computation, and if it does,the flow returns to Step S2300, and if it does not, the computationalprocessing ends.

The control unit 2740 executes the above-described operations of thearithmetic circuit 2415 for each pixel block 2131. Note that thearithmetic circuit 2415 may perform predetermined computation on pixeldata in a current frame, by using pixel data in a previous frame. Inthis case, the control unit 2740 may use, for the arithmetic circuit2415, respective added average values (Dr, Dg, Db) for the RGB pixelsfor example in a previous frame in the pixel block 2131 of itself, inplace of the respective average values of the RGB pixels of the adjacentpixel blocks 2131. The added average value of a previous frame isobtained as follows:

Di=Σ(i pixels in a pixel block of a previous frame)/(the number of ipixels in the pixel block of the previous frame)(i=r,g,b)

The control unit 2740 reads out RGB pixel data of the previous framefrom the memory 2940, and causes a fourth computing unit to calculatethe added average values (Dr, Dg, Db). Other operations are similar tothose in FIG. 58; therefore, explanation thereof is omitted.

In this manner, in the present example, computation results andcomputational contents for each pixel block 2131, and controlinformation on each pixel block 2131 by the control unit 2740 can betransmitted from the pixel block 2131 to the system control unit 2501via the output circuit 2922. As a result, the load of the imageprocessing in the system control unit 2501 can be reduced considerably.Also, because the arithmetic circuit 2415 only has to output acorrelation value with pixel data of the peripheral pixel block 2131 asan evaluation value of the pixel blocks 2131, the amount of data thatshould be transmitted to the system control unit 2501 can be reduced.Furthermore, because the arithmetic circuit 2415 in the present examplefeeds back the difference values (ΔCr, ΔCg, ΔCb) to the pixel memory2414 corresponding to the pixel block 2131, the amount of data to betransmitted to the system control unit 2501 can be reduced by acorresponding amount. Furthermore, because the image processing unit2511 included in the system control unit 2501 can generate a singlepiece of image data based on computation results received from eachoutput circuit 2922, the image processing speed can be improved ascompared with a case where RGB pixel data of all the pixel blocks 2131is stored once in the memory 2940 and read out to reconfigure a singleimage. Note that the signal processing chip 2111 in the present examplehas at least a part of functions of the image processing functions inthe image processing unit 2511. For example, the arithmetic circuit 2415further functions as an image processing unit that performs, based oneach evaluation value, image processing on image data of an image thatcorresponds to corresponding pixel signals. As one example, the imageprocessing functions may be a function of feeding back the differencevalues (ΔCr, ΔCg, ΔCb) to the pixel memory 2414. Note that examples ofthe evaluation value include an average of pixel signals within a pixelblock 2131, a weighted average of pixel signals within and outside apixel block 2131, contrast within a pixel block 2131, a weighted averageof contrast within and outside a pixel block 2131, luminance within apixel block 2131, and a weighted average of luminance within and outsidea pixel block 2131. Furthermore, the evaluation value may be a valueobtained by adding an average of G pixels, an average of R pixels, andan average of B pixels at a predetermined ratio. Also, the averagevalues may be calculated as average values of a partial area placedwithin a unit group.

FIG. 59 illustrates one example of a data array 2950 that is generatedby the output circuit 2922 based on an input from the arithmetic circuit2415. The data array 2950 has a data code area 2952 and a data area2954. Four bits for a data code may be allocated to the data code area2952. In the present example, D12 to D15 are allocated for a data code.12 bits for additional data that corresponds to each data code may beallocated to the data area 2954. In the present example, D0 to D11 areallocated for additional data. The number of bits of the data array 2950is not limited to 16 bits, but the number of bits to be allocated to thedata code and additional data may be arbitrarily set.

Note that the control unit 2740 may output computation result data fromthe arithmetic circuit 2415 via another route that is different fromthat of pixel data from the pixel memory 2414. For example, the controlunit 2740 may transmit computation results of the arithmetic circuit2415 to the system control unit 2501 through the output circuit 2922.Also, the control unit 2740 may store pixel data of the pixel memory2414 in the memory 2940 via the data transfer line 2920. In anotherexample, the control unit 2740 may attach computation results of pixeldata of the pixel block 2131 to the pixel data of the pixel block 2131,and transmit them to the system control unit 2501 from the outputcircuit 2922 altogether.

Note that although an example of calculating an average of pixel valuesis explained above, the computational contents in the arithmetic circuit2415 are not limited thereto. Parameters that are used for thearithmetic circuit 2415 may include information other than pixel values.For example, the arithmetic circuit 2415 may perform predeterminedcomputation by using parameters such as the position of a pixel in an XYplane, information about distance to a subject, a diaphragm value, anelectrical charge accumulation period in the PD 2104, an electricalcharge-voltage conversion gain in the pixel block 2131, a drive framefrequency in the pixel block 2131 (frame rate), and the like.

FIG. 60 illustrates one example of contents of the data array 2950illustrated in FIG. 59. 16 types of data codes (0 to 9, a to f) arestored in the data code area 2952. An R pixel added average value (Ar)of the pixel block 2131 is allocated to the data code 0, and is outputas 12-bit additional data. A G pixel added average value (Ag) of thepixel block 2131 is allocated to the data code 1, and is output as12-bit additional data. A B pixel added average value (Ab) of the pixelblock 2131 is allocated to the data code 2, and is output as 12-bitadditional data. A difference ΔAr between the overall average values Cr,Ar is allocated to the data code 3, and is output as 12-bit additionaldata. A difference ΔAg between the overall average values Cg, Ag isallocated to the data code 4, and is output as 12-bit additional data. Adifference ΔAb between the overall average values Cb, Ab is allocated tothe data code 5, and is output as 12-bit additional data. Theabove-described data is one example of data of computational contentsand computation results output by the arithmetic circuit 2415.

The data array 2950 additionally includes control information of thecontrol unit 2740. In the present example, an electrical charge-voltageconversion gain of the pixel block 2131 is allocated to the data code d,and is output as 12-bit additional data. A drive frame frequency of thepixel block 2131 is allocated to the data code e, and is output as12-bit additional data. An accumulation period of the pixel block 2131is allocated to the data code f, and is output as 12-bit additionaldata. By adding control information (control log) of the control unit2740 to the data array 2950, the control information indicating how thecontrol unit 2740 has controlled each pixel block 2131 can betransmitted from the pixel block side to the system control unit 2501.

That is, because the system control unit 2501 can receive the data array2950, which is exemplified in FIG. 59, for each pixel block 2131, thesystem control unit 2501 can easily execute image processing on eachpixel block 2131 by performing processing, based on the data code of thedata array 2950, on data read out by accessing respective pieces ofdifferential data of the RGB pixels of the pixel block 2131 that isstored in the memory 2940. That is, because a part of the processing inthe system control unit 2501 is performed in the arithmetic circuit2415, the load of pixel data processing on the system control unit 2501in motion image generation can be reduced considerably. Note that thesystem control unit 2501 can effectively utilize contents of the dataarray 2950 output by the output circuit 2922 while reducing the load onthe system control unit 2501 itself. For example, the system controlunit 2501 can generate a motion image by changing compression rates forrespective pixel blocks 2131 based on the contents of the data array2950.

FIG. 61 is a sectional view of another imaging element 3100 according tothe present embodiment. The imaging element 3100 includes an imagingchip 3113 that outputs a pixel signal corresponding to incident light, asignal processing chip 3111 that processes the pixel signal, and amemory chip 3112 that stores the pixel signal. These imaging chip 3113,signal processing chip 3111, and memory chip 3112 are layered, and areelectrically connected with each other via conductive bumps 3109, suchas Cu.

Note that, as illustrated, incident light is incident mainly in the Zaxis positive direction that is indicated with an outlined arrow. Inthis specification, the surface of the imaging chip 3113 on a side onwhich the incident light is incident is called a backside. Also, asindicated with coordinate axes, the leftward direction on the figurethat is orthogonal to the Z axis is referred to as the X axis positivedirection, and the front side direction in the figure that is orthogonalto the Z and X axes is referred to as the Y axis positive direction. Inseveral figures mentioned below, the coordinate axes are displayed suchthat the orientation of each figure can be known on the basis of thecoordinate axes in FIG. 61.

One example of the imaging chip 3113 is a backside illuminating type MOSimage sensor. A PD layer 3106 is disposed on a backside of aninterconnection layer 3108. The PD layer 3106 has a plurality of PDs(photo diodes) 3104 that are two-dimensionally disposed, accumulateelectrical charges according to incident light, and generate pixelsignals according to the accumulated electrical charges, and transistors3105 provided corresponding to the PDs 3104.

Color filters 3102 are provided on the incident light incidence side ofthe PD layer 3106 via a passivation film 3103. There is a plurality oftypes of the color filters 3102 that allow passage of mutually differentwavelength ranges, and the color filters 3102 are arrayed particularlycorresponding to the respective PDs 3104. The arrays of the colorfilters 3102 are described below. A set of the color filter 3102, the PD3104, and the transistor 3105 forms one pixel.

A microlens 3101 is provided, corresponding to each pixel, on theincident light incidence side of the color filter 3102. The microlens3101 condenses incident light toward the corresponding PD 3104.

The interconnection layer 3108 has interconnections 3107 that transmit apixel signal from the PD layer 3106 to the signal processing chip 3111.The interconnection 3107 may be a multilayer, and may be provided with apassive element and an active element.

A plurality of the bumps 3109 is disposed on a surface of theinterconnection layer 3108. The plurality of bumps 3109 are aligned witha plurality of the bumps 3109 that are provided on the opposing surfaceof the signal processing chip 3111, and, for example, the imaging chip3113 and the signal processing chip 3111 are pressed against each other;thereby, the aligned bumps 3109 are bonded and electrically connectedwith each other.

Similarly, a plurality of the bumps 3109 are disposed on the mutuallyopposing surfaces of the signal processing chip 3111 and the memory chip3112. These bumps 3109 are aligned with each other, and, for example,the signal processing chip 3111 and the memory chip 3112 are pressedagainst each other; thereby, the aligned bumps 3109 are bonded andelectrically connected with each other.

Note that bonding between the bumps 3109 is not limited to Cu bumpbonding by solid phase diffusion, but microbump joining by soldermelting may be adopted. Also, approximately one bump 3109 may beprovided, for example, for each pixel block described below.Accordingly, the size of the bumps 3109 may be larger than the pitch ofthe PDs 3104. Also, in a peripheral area other than a pixel area wherepixels are arrayed, a bump that is larger than the bumps 3109corresponding to the pixel area may also be provided.

The signal processing chip 3111 has a TSV (through-silicon via) 3110that connects circuits that are provided on a frontside and a backside,respectively. The TSV 3110 is preferably provided in the peripheralarea. Also, the TSV 3110 may be provided also in the peripheral area ofthe imaging chip 3113, and the memory chip 3112.

FIG. 62 is a diagram for explaining a pixel array and a pixel block 3131of the imaging chip 3113. FIG. 62 shows a state of the imaging chip 3113as observed from the backside. A matrix of a plurality of pixels isarrayed in the pixel area 3700. In FIG. 62, adjacent four pixels (fourpixels, 16 pixels, form one pixel block 3131. Grid lines in the figureshow the concept that adjacent pixels are grouped to form the pixelblock 3131. The number of pixels that form the pixel block 3131 is notlimited thereto, but may be approximately 1000, for example thirty twopixels (sixty four pixels, or more or less.

As illustrated in the partially enlarged view of the pixel area 3700,the pixel block 3131 includes, within its upper left, upper right, lowerleft, and lower right portions, four so-called Bayer arrays eachincluding four pixels including green pixels Gb, Gr, a blue pixel B, anda red pixel R. The green pixels have green filters as the color filters3102, and receive light in the green wavelength band of incident light.Similarly, the blue pixel has a blue filter as the color filter 3102,and receives light in the blue wavelength band, and the red pixel has ared filter as the color filter 3102, and receives light in the redwavelength band.

In the present embodiment, at least one pixel block is selected fromamong a plurality of the pixel blocks 3131, and pixels included in eachpixel block are controlled with control parameters that are differentfrom those for other pixel blocks. Examples of the control parametersinclude a frame rate, a thinning rate, the number of added rows whosepixel signals are added, a period or the number of times of accumulatingelectrical charges, the number of bits for digitization, and the like.Furthermore, the control parameters may be parameters in imageprocessing performed after acquiring image signals from a pixel. Theframe rate refers to a cycle of generating pixel signals. Note that inthis specification, the frame rate may refer to frame rates of therespective pixel blocks 3131. For example, a reference frame rate and ahigh frame rate refer to frame rates of the respective pixel blocks 3131

FIG. 63 is a schematic that corresponds to the pixel block 3131 of theimaging chip 3113. In the figure, a rectangle that is indicated withdotted lines representatively represents a circuit that corresponds toone pixel. Note that at least a part of each transistor explained belowcorresponds to the transistor 3105 in FIG. 61.

Although in FIG. 63, the pixel block 3131 formed with 16 pixels isillustrated, the number of pixels of the pixel block 3131 is not limitedthereto. The 16 PDs 3104 that correspond to respective pixels areconnected with respective transfer transistors 3302, and the gate ofeach transfer transistor 3302 is connected with a TX interconnection3307 to which transfer pulses are supplied. In the example illustratedin FIG. 63, the TX interconnection 3307 is connected in common to the 16transfer transistors 3302.

The drain of each transfer transistor 3302 is connected with the sourceof each corresponding reset transistor 3303, and also a so-calledfloating diffusion FD between the drain of the transfer transistor 3302and the source of the reset transistor 3303 is connected with the gateof an amplifying transistor 3304. The drain of the reset transistor 3303is connected with a Vdd interconnection 3310 to which power supplyvoltage is supplied, and its gate is connected with a resetinterconnection 3306 to which reset pulses are supplied. In the exampleillustrated in FIG. 63, the reset interconnection 3306 is connected incommon to the 16 reset transistors 3303.

The drain of each amplifying transistor 3304 is connected with the Vddinterconnection 3310 to which power supply voltage is supplied. Also,the source of each amplifying transistor 3304 is connected with thedrain of each corresponding selecting transistor 3305. The gate of eachselecting transistor is connected with a decoder interconnection 3308 towhich selection pulses are supplied. In the example illustrated in FIG.63, the decoder interconnection 3308 is provided independently to eachof the 16 selecting transistors 3305. Then, the source of each selectingtransistor 3305 is connected with a common output interconnection 3309.A load current source 3311 supplies current to the outputinterconnection 3309. That is, the output interconnection 3309 for theselecting transistors 3305 is formed by a source follower. Note that theload current source 3311 may be provided on the imaging chip 3113 sideor on the signal processing chip 3111 side.

Here, a flow from the start of electrical charge accumulation to pixeloutput after the end of the accumulation will be explained. When resetpulses are applied to the reset transistor 3303 through the resetinterconnection 3306, and simultaneously transfer pulses are applied tothe transfer transistor 3302 through the TX interconnection 3307,potential of the PD 3104 and the floating diffusion FD is reset.

When the application of the transfer pulses is stopped, the PD 3104converts received incident light into electrical charges, which are thenaccumulated. Thereafter, when transfer pulses are applied again in astate where reset pulses are not being applied, accumulated electricalcharges are transferred to the floating diffusion FD, and the potentialof the floating diffusion FD changes from reset potential to signalpotential after electrical charge accumulation. Then, when selectionpulses are applied to the selecting transistor 3305 through the decoderinterconnection 3308, variation in the signal potential of the floatingdiffusion FD is transmitted to the output interconnection 3309 via theamplifying transistor 3304 and the selecting transistor 3305. Thereby,pixel signals corresponding to the reset potential and the signalpotential are output from the unit pixel to the output interconnection3309.

In the example illustrated in FIG. 63, the reset interconnection 3306and the TX interconnection 3307 are common to the 16 pixels that formthe pixel block 3131. That is, the reset pulses and the transfer pulsesare, respectively, applied simultaneously to all the 16 pixels.Accordingly, all the pixels that form the pixel block 3131 startelectrical charge accumulation at the same timing, and end electricalcharge accumulation at the same timing. Note that however pixel signalsthat correspond to accumulated electrical charges are output selectivelyto the output interconnection 3309 upon sequential application ofselection pulses to the respective selecting transistors 3305. Also, thereset interconnection 3306, the TX interconnection 3307, and the outputinterconnection 3309 are provided separately for each pixel block 3131.

By configuring a circuit on the basis of the pixel block 3131 in thismanner, an electrical charge accumulation period can be controlled foreach pixel block 3131. In other words, adjacent pixel blocks 3131 can becaused to output pixel signals for different electrical chargeaccumulation periods. Furthermore, by causing one pixel block 3131 torepeat electrical charge accumulation several times and output a pixelsignal at each time while another pixel block 3131 is caused to performelectrical charge accumulation once, these pixel blocks 3131 can becaused to output respective frames for a motion image at different framerates.

FIG. 64A illustrates a part of a configuration of the imaging element3100, and its operation examples. The imaging element 3100 in thepresent example further has a storage unit 3114 in addition to theconfiguration illustrated in FIG. 61. Note that the storage unit 3114may be provided to the signal processing chip 3111. In this case, theimaging element 3100 does not have to have the memory chip 3112. Also,the storage unit 3114 may be provided to the memory chip 3112.

The imaging chip 3113 has an pixel area 3700 in which a plurality ofpixels that respectively generate pixel signals according to incidentlight are placed. Here, the pixel area 3700 may be configured byarranging a plurality of pixels two-dimensionally. Each pixel block 3131has m (n pixels in the row and column directions. Here, m and n areintegers that are equal to or larger than two. The pixel area 3700 has aplurality of the pixel blocks 3131 that are divided in the row andcolumn directions. As illustrated in FIG. 62, the pixel block 3131refers to an aggregate of pixels in which a plurality of pixels areplaced in a matrix. Also, the row and column directions refer to twodifferent directions in the plane of the pixel area 3700, and may notnecessarily be orthogonal to each other.

Although, for the sake of convenience of explanation, three (in the rowdirection) (three (in the column direction) pixel blocks 3131 areindicated in FIGS. 64A to 64C, the number of the pixel block 3131included in the pixel area 3700 may be larger. The numbers of pixelsincluded in each pixel block 3131 are preferably the same. Also, thenumber of pixels included in each pixel block 3131 within the pixel area3700 is fixed. The pixel block 3131 is configured with, for example, 32(64 pixels.

The signal processing chip 3111 in the present example has, for eachpixel block 3131, a multiplexer 3411, an A/D converter 3412, ade-multiplexer 3413 and a control unit 3740. The multiplexer 3411sequentially selects pixels included in the corresponding pixel block3131, and inputs pixel signal corresponding to the selected pixels tothe A/D converter 3412. The A/D converter 3412 converts analog pixelsignals into digital pixel data, and inputs it to the de-multiplexer3413. The de-multiplexer 3413 causes a storage area corresponding to thepixel to store the pixel data in a corresponding storage block 3730. Therespective storage blocks 3730 pass the stored pixel data over to thearithmetic circuit in the subsequent step.

The storage unit 3114 is provided corresponding to a plurality of pixelblocks 3131, and has a plurality of the storage blocks 3730 that canstore pixel data of respectively corresponding pixel blocks 3131. Thestorage block 3730 corresponds one-to-one to the pixel block 3131. Thestorage block 3730 may be connected with the corresponding pixel block3131 via a bus 3720. The storage block 3730 may be a buffer memory.

Also, at least a part of the storage block 3730 can store pixel data ofa pixel block other than the corresponding pixel block 3131. That is, asingle storage block 3730 may be shared by a plurality of the pixelblocks 3131. In other words, the control unit 3740 can cause pixel dataof a single pixel block 3131 to be stored in a plurality of the storageblocks 3730. Because a plurality of the storage blocks 3730 can beutilized efficiently by sharing the storage blocks 3730 as describedbelow, the memory capacity of the entire storage unit 3114 can besuppressed.

Note that about all the pixel blocks 3131, preferably pixel data can bewritten in and read from at least one other storage block 3730 otherthan the corresponding storage block 3730. The other storage block 3730may be predetermined for each pixel block 3131, or may be dynamicallychangeable. Also about all the storage blocks 3730, preferably pixeldata is written in and read from at least one other pixel block 3131other than the corresponding pixel block 3131. The other pixel block3131 may be predetermined for each storage block 3730, or may bedynamically changeable.

Note that each storage block 3730 may be a memory that is provided toeach pixel block 3131 in an area of the signal processing chip 3111 thatoverlaps with a corresponding pixel block 3131. That is, the storageblock 3730 may be provided in an area immediately below a correspondingpixel block 3131 in the signal processing chip 3111. In this case, thepixel block 3131 and the storage block 3730 may be electricallyconnected via TSV. Also, the corresponding storage block 3730, A/Dconverter 3412 and the like are provided in an area of the signalprocessing chip 3111 that overlaps with each pixel block 3131. Also,each storage block 3730 may be a memory that is provided outside an areaof the signal processing chip 3111 that overlaps with the pixel area3700.

Also, when the respective storage block 3730 and A/D converter 3412 isprovided in an area that overlaps with a corresponding pixel block 3131,and when the respective storage block 3730 stores pixel data of a pixelblock 3131 other than the corresponding pixel block 3131, an analogpixel signal or digital pixel data may be transmitted to an area wherethe storage block 3730 is provided. In the former case, the A/Dconverter 3412 that corresponds to the storage block 3730 converts thepixel signal into pixel data, and inputs it to the storage block 3730.In the latter case, the pixel signal is converted into pixel data in theA/D converter 3412 in the area that overlaps with the pixel block 3131,and then the pixel data is transmitted to a storage block 3730 where thepixel data should be stored. Interconnections for transmitting thesepixel signals or pixel data are provided in the signal processing chip3111.

FIG. 64B illustrates another operation example of the imaging element3100. Note that the configuration of the signal processing chip 3111illustrated in FIG. 64A is omitted in FIG. 64B. In the present example,pixel data of a pixel block 3712 among a plurality of pixel blocks 3131is stored in any of other storage blocks 3731, 3732, 3733 other than acorresponding storage block 3734. In the present example, analog pixelsignals generate by the pixel block 3712 are converted into digitalpixel data by the A/D converter 3412 that corresponds to the otherstorage block 3731 to 3733. In this manner, the use efficiency of amemory can be improved by making pixel data of any pixel block 3712storable in a plurality of the storage blocks 3731 to 3734.

For example, a plurality of the pixel blocks 3131 may generate pixelsignals of a subject imaged at different frame rates among respectivepixel blocks 3131, at timing according to the frame rates. As describedbelow, the control unit 3740 selects a corresponding storage block 3730for each pixel block 3131 from among at least two frame rates that are areference frame rate and a high frame rate whose cycle is shorter thanthat of the reference frame rate. The cycle of the high frame rate maybe a multiple of 1/an integer of the cycle of the reference frame rate.Each pixel block 3131 may output pixel signals that correspond to oneblock at each cycle of the frame rate.

In the present example, a case where the frame rate of the pixel block3712 is quintuple of the reference frame rate is explained. Also,substantially simultaneous with output of pixel signals by the referenceframe rate pixel block 3131, the pixel block 3712 of the high frame ratealso outputs pixel signals. In this case, the pixel block 3712 outputspixel signals four times until the pixel block 3131 outputs next pixelsignals.

When the reference frame rate pixel block 3131 is not outputting pixelsignals, the control unit 3740 causes pixel data according to pixelsignals that correspond to four times in which the pixel block 3712 ofthe high frame rate output pixel signals to be stored in the pluralityof storage block 3731 to 3734. Note that pixel data that corresponds toone frame according to pixel signals output by each pixel block insynchronization with the reference frame rate may be stored in a memorythat is different from the plurality of storage blocks 3730, and afterthe pixel data is once stored in the plurality of storage blocks 3730,and before next pixel data of the pixel block 3712 that operates at thehigh frame rate is input to the plurality of storage blocks 3730, may bepassed over to a memory or a circuit in the subsequent step of thestorage block 3730. Thereby, a plurality of storage blocks can be usedefficiently.

When pixel data has already been stored in a storage block 3734 thatcorresponds to the pixel block 3712 of the high frame rate, the controlunit 3740 causes the pixel data that corresponds to the pixel block 3712to be stored in any of the storage blocks 3731, 3732, 3733 in whichpixel data has not stored yet. That is, the control unit 3740 causespixel data of the pixel blocks 3712 of the high frame rate to beallocated to and stored in the storage blocks 3731, 3732, 3733, in whichpixel data has not been stored, other than the corresponding storageblock 3734. At this time, the pixel data to be allocated may have, asadditional data, positional data of a pixel block 3712 that correspondsto the pixel data in the pixel area 3700, and frame data that indicatesa frame to which the pixel data belongs. The position of the storageblock 3730 to which the pixel data should be allocated may be fixed foreach pixel block 3712 or may vary dynamically. When the position of thestorage block 3730 to which pixel data should be allocated is fixed foreach pixel block 3131, positional data can be omitted from theadditional data.

FIG. 64C illustrates another operation example of the imaging element3100. Note that the configuration of the signal processing chip 3111illustrated in FIG. 64A is omitted in FIG. 64C. In the present example,as in the example of FIG. 64B, pixel data of the pixel block 3712 isstored in any of other storage blocks 3735 to 3738 other than thecorresponding storage block 3734. Note that however in the presentexample, pixel signals are converted into pixel data by the A/Dconverter 3412 in an area that overlaps with the pixel block 3712, andthen are transmit to storage blocks where the pixel data should bestored. In the present example, the pixel data moves between storageblocks.

The control unit 3740 in the present example is different from that inthe embodiment illustrated in FIG. 64B in that in the present example,when pixel data has already been stored in the storage block 3734 thatcorresponds to the pixel block 3712 of the high frame rate, the pixeldata of the storage block 3734 is moved to the storage blocks 3735,3736, 3737, 3738 in which pixel data has not been stored, and is storedin the respective storage blocks. That is, in the present example,storage blocks are connected with each other by interconnections suchthat the storage unit 3114 is able to transmit and receive data betweenthe storage blocks.

In the control unit 3740, pixel data of the storage block 3734 is movedto and stored in any of the storage blocks 3735, 3736, 3737, 3738 inwhich pixel data has not been stored. Preferably, the control unit 3740may move pixel data of the storage block 3734 toward a storage blockthat corresponds to the pixel block 3131 at the outermost circumferenceof the pixel area 3700 and store the pixel data therein. Because in manycases, the frame rate of the pixel block 3131 becomes lower at aposition farther from the high frame rate pixel block 3712 toward theouter circumferential direction of the pixel area 3700, the control unit3740 preferably distributes pixel data two-dimensionally in the outercircumferential direction. In this manner, by utilizing the plurality ofstorage blocks 3730 evenly, the memory capacity of the entire storageunit 3114 can be suppressed without increasing the capacity of a buffermemory. Note that the control unit 3740 may select a storage block 3730that corresponds to a pixel block 3131 not positioned in the outermostcircumference based on frame rate information on each pixel block 3131,and write the pixel data therein.

In the present example also, the position of a storage block to whichpixel data should be distributed may be fixed or may vary dynamically.When the position of a storage block to which pixel data should bedistributed is fixed, positional data can be omitted from the additionaldata which should be added to the moved pixel data. In this case, astorage block to which pixel data should be distributed is preferably astorage block that corresponds to the pixel block 3131 at the outermostcircumference of the pixel area 3700. Also, pixel data stored in eachstorage block 3730 may be sequentially moved in synchronization with ahigh frame rate. Thereby, the pixel data can be transmitted between thestorage blocks 3730 that are spaced apart. By repeatedly moving thepixel data, the pixel data can be moved to a given storage block 3730.

An arithmetic circuit 3415 described below processes the pixel datastored in the storage block 3730, and passes it over to an imageprocessing unit in the subsequent step. The arithmetic circuit 3415 maybe provided in the signal processing chip 3111 or the storage unit 3114.Note that although, in the figure, connections for a single pixel block3131 are illustrated, connections actually exist for each pixel block3131, and operate in parallel. Note that however the arithmetic circuit3415 may not exist for each pixel block 3131, and, for example, a singlearithmetic circuit may sequentially perform processing by sequentiallyreferring to values of the storage blocks 3730 that correspond to therespective pixel blocks 3131.

As described above, the output interconnection 3309 is providedcorresponding to each of the pixel blocks 3131. Because the imagingelement 3100 is formed by layering the imaging chip 3113, the signalprocessing chip 3111, and the storage unit 3114, the outputinterconnection 3309 can be routed without increasing the size of eachchip in the plane direction by using inter-chip electrical connectionsthat use the bumps 3109 for the interconnection.

Note that rate information on the frame rate of each pixel block 3131 isprovided to the control unit 3740. The control unit 3740 selects astorage block 3730 that should store pixel data of the high frame ratepixel block 3131 based on the rate information. For example, the controlunit 3740 selects the storage block 3730 that corresponds to thereference frame rate pixel block 3131 as the storage block 3730 thatshould store the pixel data. Also, the control unit 3740 may decide aroute for moving pixel data in the form illustrated in FIG. 64C based onthe rate information. For example, when moving pixel data of eachstorage block 3730, the control unit 3740 selects, among storage blocks3730 that are adjacent to the storage block 3730 and correspond to areference frame rate, a storage block 3730 whose distance from thestorage block 3730 that corresponds to a high frame rate is larger.

FIG. 65 is a block diagram illustrating a configuration of an imagingdevice according to the present embodiment. An imaging device 3500includes an imaging lens 3520 as an imaging optical system, and theimaging lens 3520 guides a subject luminous flux that is incident alongan optical axis OA to the imaging element 3100. The imaging lens 3520may be a replaceable lens that can be attached/detached to and from theimaging device 3500. The imaging device 3500 includes, mainly, theimaging element 3100, a system control unit 3501, a drive unit 3502, aphotometry unit 3503, a work memory 3504, a recording unit 3505, and adisplay unit 3506.

The imaging lens 3520 is configured with a plurality of optical lensgroups, and forms an image of a subject luminous flux from a scene nearits focal plane. Note that, in FIG. 61, the imaging lens 3520 isrepresentatively shown with a single virtual lens that is placed nearthe pupil. The drive unit 3502 is a control circuit that executeselectrical charge accumulation control such as timing control and areacontrol on the imaging element 3100 according to instructions from thesystem control unit 3501. In this sense, it can be said that the driveunit 3502 serves functions of an imaging element control unit thatcauses the imaging element 3100 to execute electrical chargeaccumulation and output pixel signals.

The imaging element 3100 passes pixel signals over to an imageprocessing unit 3511 of the system control unit 3501. The imageprocessing unit 3511 performs various types of image processing by usingthe work memory 3504 as a workspace, and generates image data. Forexample, when image data in a JPEG file format is generated, compressionprocesses are executed after color video signals are generated fromsignals obtained from Bayer arrays. The generated image data is recordedin the recording unit 3505 and converted into display signals, and isdisplayed on the display unit 3506 for a preset period of time.

The photometry unit 3503 detects luminance distribution of a scene priorto an imaging sequence for generating image data. The photometry unit3503 includes an AE sensor of approximately one million pixels, forexample. A computing unit 3512 of the system control unit 3501calculates luminance of respective areas within a scene, upon receivingan output of the photometry unit 3503. The computing unit 3512 decides ashutter speed, a diaphragm value, and an ISO speed according to thecalculated luminance distribution. The imaging element 3100 may doubleas the photometry unit 3503. Note that the computing unit 3512 executesvarious types of computation for operating the imaging device 3500.

The drive unit 3502 may be partially or entirely mounted on the imagingchip 3113, or partially or entirely mounted on the signal processingchip 3111. The system control unit 3501 may be partially mounted on theimaging chip 3113 or the signal processing chip 3111.

FIG. 66 is a functional block diagram of the image processing unit. Theimage processing unit 3511 in the present example extracts the pixelblock 3131 that operates at a reference frame rate (a peripheral area3176 described below) and the pixel block 3131 that operates at a highframe rate (an attention area 3172 described below). The imageprocessing unit 3511 has, in addition to the above-described functions,a subject estimating unit 3150, a group selecting unit 3152, a motionimage generating unit 3154, and a motion image synthesizing unit 3156.Each of these functions is described below.

FIG. 67 is a flowchart that illustrates operations of an imaging deviceto generate and record a motion image. FIGS. 68 and 69 each illustrateone example of an image imaged by an imaging element. FIG. 70illustrates a relationship between respective frame rates and outputtiming of image signals.

Operations in FIG. 67 start when a user instructs the imaging device3500 to generate a motion image for example by pressing down a recordbutton. First, the subject estimating unit 3150 drives the drive unit3502 to acquire image data based on image signals from the imagingelement 3100, and estimate a main subject included in an image indicatedby the image data (S3100).

In this case, the drive unit 3502 preferably causes image signals frompixel blocks 3131 included in an entire imaging area, for example allthe pixel blocks 3131, to be output. Also, the drive unit 3502 may causeimage signals from all the pixels included in each pixel block 3131 tobe output, or causes image signals from pixels that are thinned at apredetermined thinning rate to be output. The subject estimating unit3150 compares a plurality of images obtained from the imaging element3100 in a time-series, and identifies a moving subject as a mainsubject. Note that another method may be used to estimate a mainsubject.

For example, when the subject estimating unit 3150 acquires an image3170 in FIG. 68 and an image 3178 in FIG. 69 from the imaging element3100 as temporally sequential images, based on differences therebetween,the subject estimating unit 3150 identifies a child as a main subject3171. Note that grid lines in the image 3170 and the image 3178 indicateboundaries of the pixel blocks 3131, but the number of the pixel blocks3131 is merely an example, and is not limited to the number shown in thefigures.

The group selecting unit 3152 selects at least one pixel block 3131 onwhich image light of the main subject 3171 estimated by the subjectestimating unit 3150 is incident (S3102). For example, pixel blocks 3131including at least a part of the main subject 3171 are selected in theimage 3170. Furthermore, considering that the main subject 3171 moves inan imaging area, the group selecting unit 3152 preferably selects pixelblocks 3131 that further surround the pixel blocks 3131 including atleast a part of the main subject 3171.

The group selecting unit 3152 handles a set of these selected pixelblocks 3131 as an attention area 3172. Furthermore, the group selectingunit 3152 handles, as a peripheral area 3176, a set of pixel blocks 3131not included in the attention area 3172 in the entire imaging area. Thegroup selecting unit 3152 identifies area information 3174 thatindicates a range of the attention area 3172 in relation to the entireimaging area.

In the example illustrated in FIG. 68, the attention area 3172 is arectangular area including total 28 pixel blocks 3131 (seven in thehorizontal direction (four in the vertical direction). On the otherhand, the peripheral area 3176 includes 98 pixel blocks 3131 excludingthe attention area 3172 from total 126 pixel blocks 3131 (21 in thehorizontal direction (six in the vertical direction) which constitutethe imaging area. Also, the position (9, 2) of the attention area 3172in the imaging area that is counted from the left side and the upperside of the upper left end pixel block 3131 in the figure is identifiedas the area information 3174. Furthermore, the numbers in the horizontaland vertical directions, 7 (4, of the attention area 3172 are identifiedas size information.

The group selecting unit 3152 transmits information for identifying thepixel blocks 3131 included in the attention area 3172, and informationfor identifying the peripheral area 3176 to the drive unit 3502. In thiscase, information on frame rates to be applied to the attention area3172 and the peripheral area 3176, respectively, is transmittedtogether. Here, the frame rate to be applied to the attention area 3172is preferably higher than the frame rate to be applied to the peripheralarea 3176. For example, when the frame rate to be applied to theperipheral area 3176 is 60 fps, the frame rate to be applied to theattention area 3172 is set to 180 fps. Preferably, values of the framerates are preset, and stored such that the group selecting unit 3152 canrefer to them, but may be changeable with an operation of a userafterwards.

The drive unit 3502 drives the imaging element 3100 to perform imagingat the respective frame rates (S3104). That is, the drive unit 3502causes the pixel blocks 3131 included in the attention area 3172 toexecute electrical charge accumulation and image signal output at a highframe rate, and causes the pixel blocks 3131 included in the peripheralarea 3176 to execute electrical charge accumulation and image signaloutput at a low frame rate. In other words, the drive unit 3502 obtainsimage signals that correspond to a plurality of frames that arecontiguous in a time-series for the pixel blocks 3131 included in theattention area 3172 while obtaining image signals that correspond to asingle frame for the pixel blocks 3131 included in the peripheral area3176.

For example, when the frame rate of the peripheral area 3176 is set to60 fps and the frame rate of the attention area 3172 is set to 180 fps,as illustrated in FIG. 70, the drive unit 3502 obtains image signals ofthree frames A1, A2, A3 from the attention area 3172 during time 1/60 sin which image signals of a single frame B1 from the peripheral area3176 are obtained ( 1/60 s=3× 1/180 s). In this case, the drive unit3502 obtains image signals at different frame rates by separatelydriving a set of the reset transistors 3303, the transfer transistors3302, and the selecting transistors 3305 of the pixel blocks 3131included in the peripheral area 3176, and a set of the reset transistors3303, the transfer transistors 3302, and the selecting transistors 3305of the pixel blocks 3131 included in the attention area 3172.

Note that FIG. 70 illustrates timing of outputting image signals, butdoes not illustrate length of an exposure period. The drive unit 3502drives the above-described sets of the transistors for the peripheralarea 3176 and for the attention area 3172 such that the exposure periodpreviously calculated by the computing unit 3512 can be attained.

In addition to this, the length of the exposure period may be changedaccording to frame rates. For example, in the example illustrated inFIG. 70, the exposure period of one frame of the peripheral area 3176may be set to ⅓, which is substantially the same with that for theattention area 3172. Also, image signals may be corrected by a ratio offrame rates after outputting the image signals. Also, the timing ofoutputting image signals may not be synchronous as in FIG. 70, but maybe asynchronous between the peripheral area 3176 and the attention area3172.

The image processing unit 3511 sequentially stores, on a frame-by-framebasis, image signals from the attention area 3172 in a predeterminedstorage area of the work memory 3504 (S3106). Similarly, the imageprocessing unit 3511 sequentially stores, on a frame-by-frame basis,image signals from the peripheral area 3176 in a predetermined storagearea of the work memory 3504 (the same step). The work memory 3504 has aplurality of storage blocks 3730 as explained in FIGS. 64A to 64C. Thework memory 3504 may be a memory that includes a memory group thatcorresponds to each pixel block 3131.

The motion image generating unit 3154 reads out the image signals of theattention area 3172 stored in the work memory 3504 (S3108), andgenerates data of the attention area motion image which includes aplurality of frames of the attention area 3172 (S3110). Similarly, themotion image generating unit 3154 reads out the image signals of theperipheral area 3176 stored in the work memory 3504, and generates dataof the peripheral area motion image which includes a plurality of framesof the peripheral area 3176 (the same step). Here, the attention areamotion image and the peripheral area motion image may each be generatedin general-purpose formats such as MPEG and be able to be reproducedseparately, or may each be generated in dedicated formats that do notallow reproduction without going through synthesis processing describedbelow.

FIG. 71 schematically illustrates an attention area motion image and aperipheral area motion image generated by the motion image generatingunit. The motion image generating unit 3154 generates the attention areamotion image at a frame rate that corresponds to a frame rate at whichthe drive unit 3502 drove the attention area 3172. In the exampleillustrated in FIG. 71, the attention area motion image is generated atthe frame rate 1/180 fps which is the same with the frame rate 1/180 fpsat which the drive unit 3502 drove the attention area 3172.

Similarly, the motion image generating unit 3154 generates theperipheral area motion image at a frame rate that corresponds to a framerate at which the drive unit 3502 drove the peripheral area 3176. In theexample illustrated in FIG. 71, the peripheral area motion image isgenerated at the frame rate 1/60 fps which is the same with the framerate 1/60 fps at which the drive unit 3502 drove the peripheral area3176. Note that effective values do not exist in an area of theperipheral area motion image that corresponds to the attention area3172, and the area is indicated with diagonal lines in the figure.

Furthermore, the motion image generating unit 3154 adds headerinformation to the attention area motion image and the peripheral areamotion image, and records the data in the recording unit 3505 (S3112).The header information includes the area information that indicates theposition of the attention area 3172 in relation to the entire imagingarea, the size information that indicates the size of the attention area3172, and timing information that indicates a relationship betweenoutput timing of image signals of the attention area 3172 and outputtiming of image signals of the peripheral area 3176.

The system control unit 3501 determines whether to perform imaging for anext unit time (S3114). Whether to perform imaging of a next unit timeis determined based on whether, at the time point, a user is pressingdown a motion image record button. When imaging is to be performed for anext unit time (S3114: Yes), the flow returns to the above-describedStep S3102, and when imaging is not to be performed for the next unittime (S3114: No), the operation ends.

Here, the “unit time” is preset in the system control unit 3501, andlasts for several seconds. The storage capacity used for storage at StepS3106 is determined based on this unit time, the frame rate and numberof pixel blocks of the attention area 3172, and the frame rate andnumber of pixel blocks of the peripheral area 3176. Based also on thesepieces of information, an area of the storage capacity that stores dataof the attention area 3172 and an area of the storage capacity thatstores data of the peripheral area 3176 are determined.

In this manner, image signals can be obtained at a high frame rate fromthe attention area 3172 including the main subject 3171, and also a dataamount can be reduced by keeping the frame rate for the peripheral area3176 low. Accordingly, as compared with high speed readout from all thepixels, loads of driving and image processing can be reduced, and powerconsumption and heat generation can be suppressed.

Note that when a next unit time starts in the example illustrated inFIG. 67, pixel blocks 3131 are selected again at Step S3102, and thearea information and the size information are updated. Thereby, theattention area 3172 can be updated successively by tracking the mainsubject 3171. In the example illustrated in FIG. 71, in a first frame A7of the unit time in the attention area motion image, an attention area3182 including pixel blocks 3131 that are different from those of a lastframe A6 in the previous unit time are selected, and in accordance withthis, area information 3184 and a peripheral area 3186 are updated.

FIG. 72 illustrates one example of the header information added by themotion image generating unit. The header information in FIG. 72 includesattention area motion image IDs that identify attention area motionimages, frame rates of the attention area motion images, peripheral areamotion image IDs that identify peripheral area motion imagescorresponding to the attention area motion images, frame rates of theperipheral area motion images, timing information, area information, andsize information. These pieces of the header information may be added asthe header information to either one or both of the attention areamotion image and the peripheral area motion image.

FIG. 73 is a flowchart that illustrates operations of an imaging deviceto reproduce and display a motion image. The operations start when auser specifies any of attention area motion images displayed asthumbnails on the display unit 3506, and presses down a reproductionbutton.

The motion image synthesizing unit 3156 reads out, from the recordingunit 3505, data of an attention area motion image specified by the user(S3150). The motion image synthesizing unit 3156 reads out, from therecording unit 3505, data of a peripheral area motion imagecorresponding to the attention area motion image (S3152).

In this case, the motion image synthesizing unit 3156 identifies theperipheral area motion image based on a peripheral area motion image IDindicated in the header information of the attention area motion imageread out at Step S3150. Instead of this, a peripheral area image thatincludes, as the header information, timing information which is thesame with the timing information indicated in the header information ofthe attention area motion image may be searched for and identified.

Note that the header information is included in the attention areamotion image in the above-described example. On the other hand, when theheader information is not included in the attention area motion image,but in the peripheral area motion image, the user may be, previously atStep S3150, caused to specify the peripheral area motion image which isto be read out, and the attention area motion image is specified andread out from the header information at Step S3152.

The motion image synthesizing unit 3156 synthesizes a frame of theattention area motion image and a frame of the peripheral area motionimage into a frame of a displayed motion image (S3154). In this casefirst, the first frame A1 of the attention area motion image is fittedat a position indicated by the area information 3174 in the first frameBI of the peripheral area motion image to form a synthesized first frameC1 of the displayed motion image. As illustrated in FIG. 71, the motionimage synthesizing unit 3156 causes the first frame C1 of the displayedmotion image to be displayed on the display unit 3506 (S3156).

The motion image synthesizing unit 3156 determines whether there is anext frame of the attention area motion image before a next frame B2 ofthe peripheral area motion image (S3158). When there is a next frame ofthe attention area motion image (S3158: Yes), the motion imagesynthesizing unit 3156 updates the attention area 3172 by using the nextframes A2, A3, and keeps the peripheral area 3176 at the previous frameB1 (S3162) to form next synthesized frames C2, C3 of the displayedmotion image (S3162), and display them sequentially (S3156).

On the other hand, when there is not a next frame of the attention areamotion image before the next frame B2 of the peripheral area motionimage at Step S3158 (S3158), the motion image synthesizing unit 3156updates the attention area 3172 by using a next frame A4 and updatesalso the peripheral area 3176 by using the next frame B2 (S3164) to forma next synthesized frame C4 of the displayed motion image (S3162), anddisplay it (S3156).

As long as there is a next frame of the peripheral area 3176 in theperipheral area motion image (S3160: Yes), Steps S3154 to S3160 arerepeated. When there is not a next frame of the peripheral area 3176 inthe peripheral area motion image (S3160: No), the motion imagesynthesizing unit 3156 makes a search to determine whether, at a unittime next to the unit time of the set of the attention area motion imageand the peripheral area motion image, there is a set of an attentionarea motion image and a peripheral area motion image (S3166). Forexample, the motion image synthesizing unit 3156 makes a search in thesame folder of the recording unit 3505 to determine whether there isanother attention area motion image whose header information includestiming information indicating timing that immediately follows timingindicated by timing information of the previous attention area motionimage.

As long as there is a set of an attention area motion image and aperipheral area motion image in a next unit time (S3166: Yes), StepsS3150 to S3166 are repeated. When there is not a set of an attentionarea motion image and a peripheral area motion image in a next unit time(S3166: No), the operation ends.

In this manner, a smooth motion image can be displayed about theattention area 3172 in which the main subject 3171 is included whilereducing the overall data amount. Note that although at Step S3162, theattention area 3172 is updated directly by using the next frames to formthe synthesized frames of the displayed image, the method of synthesisis not limited thereto. As another example, the boundary line of themain subject 3171 in the attention area 3172 may be identified by imageprocessing, the main subject 3171 surrounded by the boundary line may beupdated with a next frame, and the outside of the boundary line of themain subject 3171 may be kept at the previous frame even if it is withinthe attention area 3172, to form a synthesized frame with the peripheralarea 3176. That is, the frame rate of the outside of the boundary linein the attention area 3172 may be lowered to the frame rate of theperipheral area 3176. Thereby, it is possible to prevent boundaries ofsmoothness in the displayed motion image from looking unnatural. Also,the frame rates of reproduction need not be the same with the framerates at the time of imaging (180 fps for the attention area, and 60 fpsfor the peripheral area), but the frame rates may be for example 60 fpsand 20 fps for the attention area and the peripheral area, respectively.In such a case, the reproduction is slow-motion reproduction.

FIG. 74 shows a plan view of a configuration of the pixel area 3700 ofthe imaging element 3100, and its operation example. Note that FIGS. 74to 77 show figures in which each pixel block 3131 in the pixel area 3700and each storage block 3730 in the storage unit 3114 are projected ontothe same plane. Each pixel block 3131 is placed by being spaced apart atcertain intervals in the row and column directions throughout the pixelarea 3700. The pixel block 3131 has m (n pixel, and n and m are two orlarger. The pixel block 3131 may be configured with 32 (64 pixels thatare placed in a matrix. In the present example, each storage block 3730is a memory that is provided to each pixel block 3131. That is, eachpixel block 3131 has a storage block 3730 that corresponds to each otheron a one-to-one basis. Each storage block 3730 is provided in an area ofthe signal processing chip 3111 that overlaps with a corresponding pixelblock 3131.

Each pixel block 3131 is grouped into a group of a plurality of pixelblocks 3131 that are distributed being spaced apart at certain intervalsin the pixel area 3700. A storage block 3730 that corresponds to pixelblocks 3131 in a group is shared by the pixel blocks 3131 within thegroup. Being shared means that pixel data of a plurality of the pixelblock 3131 can directly or indirectly read from and write in the storageblock 3730. All the pixel blocks 3131 included in the pixel area 3700are preferably grouped such that distances among the pixel blocks 3131within a single group are maximized. Also, a group of the pixel blocks3131 more preferably includes a plurality of pixel blocks 3131 that arepositioned at the outermost circumference of the pixel area 3700 in theimaging chip 3113. In this case, the control unit 3740 controls theplurality of the pixel blocks 3131 that are positioned at the outermostcircumference at a fixed frame rate which is lower than the high framerate (in the present example, the reference frame rate).

Here, the position of the pixel block 3131 is expressed with acoordinate (x, y). In the present example, four pixel blocks 3131provided at the positions (4, 4), (4, 1), (1, 4), (1, 1) are grouped.Other pixel blocks 3131 are similarly grouped with pixel blocks 3131that are spaced apart at certain intervals.

Each storage block 3730 corresponding to the pixel blocks 3131 in agroup is shared by all the pixel blocks 3131 within the group. Thereby,pixel data of the high frame rate pixel block 3131 can be stored instorage blocks 3730 that corresponds to the reference frame rate pixelblocks 3131 within the group. In the present example, pixel data of thehigh frame rate pixel block 3131 at the position (4, 4) indicated withdiagonal lines are stored sequentially in the reference frame ratestorage blocks 3730 among the storage blocks 3730 that correspond to thepixel blocks 3131 at the positions (4,4), (4,1), (1,4), (1,1).

That is, when pixel data has already been stored in the storage block3730 that corresponds to the high frame rate pixel blocks 3131, thecontrol unit 3740 causes the pixel data that corresponds to the pixelblock 3131 to be stored in any storage block 3730 in the same group withthe pixel block 3131. Here, as illustrated in FIG. 68, the attentionarea 3172 is formed with pixel blocks 3131 that are continuously placed.Accordingly, by grouping a plurality of pixel blocks 3131 that aredistributed being spaced apart at certain intervals within the pixelarea 3700, the possibility of high frame rate pixel blocks 3131 andreference frame rate pixel blocks 3131 coexisting within a group can beincreased. By doing so, the use efficiency of a memory can be improvedwithout increasing the memory capacity of the storage blocks 3730. Also,because a group that shares a storage block 3730 is fixed, theadditional data that indicates to which pixel block 3131 pixel datastored by each storage block 3730 corresponds can be reduced or omitted.

FIG. 75 is a plan view of one example of other configurations of theimaging element 3100 illustrated in FIG. 74. The imaging element 3100 inthe present example is different from the embodiment described in FIG.74 in that the imaging element 3100 has storage units 3810 providedoutside the pixel areas 3700 and along its sides in the row and columndirections respectively, in place of the storage unit 3114. Note thatthe storage units 3810 may be the same with the storage unit 3114 inrespects other than their physical positions.

The storage unit 3810 in the present example is configured with aplurality of storage areas 3812 that are provided opposing to an areathat overlaps with the pixel blocks 3131 at the outermost circumferenceof the pixel area 3700 in the row and column directions. The respectivestorage areas 3812 are configured with two (two storage blocks 3730. Therespective storage blocks 3730 are storage areas 3812 in a memoryprovided to each group. The control unit 3740 generates addressinformation based on information about the position, frame rate, andtiming of the respective grouped pixel blocks 3131, and sequentiallywrites the pixel data in the storage blocks 3730.

In the present example, the storage block 3730 that corresponds to thegrouped pixel block 3131 configures the two (two storage area 3812. Thatis, because the storage blocks 3730 that correspond to the grouped pixelblocks 3131 are organized to be adjacent with each other at one place,it is not necessary to connect, via interconnections, the storage blocks3730 that are spaced apart as in a case where the storage blocks 3730are provided respectively to areas that overlap with the pixel blocks3131. Accordingly, long time that is required to write in/read out pixeldata due to RC delay is no longer necessary. Also, when inputting pixeldata into an arithmetic circuit in a next step, a single bus only has tobe provided to the storage area 3812. Furthermore, as compared to a casewhere storage blocks are each provided to respective areas that overlapwith the pixel blocks 3131, a circuit configuration necessary forwriting in/reading out pixel data can be simplified.

FIG. 76 is a plan view showing another operation example of the imagingelement 3100 illustrated in FIG. 74. The present example is differentfrom the embodiment illustrated in FIG. 74 in that, in the presentexample, transmission paths 3710 that transmit pixel data among thestorage blocks 3730 corresponding to adjacent pixel blocks 3131 arefurther provided. The transmission paths 3710 may be interconnectionsfor connecting between the respective storage blocks 3730. Thetransmission paths 3710 connect the control unit 3740 and all thestorage blocks 3730. The control unit 3740 sequentially moves pixel datathat corresponds to the high frame rate pixel block 3131 to the adjacentstorage blocks 3730 in synchronization with the high frame rate. Here,“in synchronization with the high frame rate” means that a plurality ofthe adjacent storage blocks 3730 is caused to sequentially store pixeldata at timing that is the same as the timing at which the high framerate pixel block 3131 takes in pixel data.

Here, it is assumed in the following example that the frame rate of thepixel block 3131 at the position (4, 4) is five-fold of the referenceframe rate. When the reference frame rate is 60 fps, the high frame rateis 300 fps. The imaging timing at the high frame rate is as follows: Ata clock time t=0, the timing is T₀; at a clock time t=1/300 s, thetiming is T₁; at a clock time t=2/300 s, the timing is T₂; at a clocktime t=3/300 s, the timing is T₃; at a clock time t=4/300 s, the timingis T₄; and at a clock time t=5/300 s, the timing is T₅.

At the timing T₀, the control unit 3740 causes the storage blocks 3730that correspond respectively to all the pixel blocks 3131 to store pixeldata of a subject imaged. Next, at the timing T₁, the control unit 3740moves the pixel data that is stored in an adjacent lower frame ratestorage block 3730 at the position (3, 4) to a storage block 3730 at theposition (2, 4) in the outer circumferential direction, and moves thepixel data that is stored in a storage block 3730 that corresponds tothe pixel block 3131 at the position (4, 4) to the storage block 3730 atthe position (3, 4) now in an empty state, and stores the pixel datatherein. Simultaneously, the control unit 3740 causes pixel data of thepixel block 3131 at the position (4, 4) acquired at the timing T₁ to bestored in the storage block 3730 at the corresponding position (4, 4).

At the timing T₂, the control unit 3740 moves the pixel data that isstored in a storage block 3730 at the position (4, 3) to a storage block3730 at the position (4, 2) in the outer circumferential direction, andstores the pixel data therein, and moves the pixel data of the storageblock 3730 that corresponds to the pixel block 3131 at the position (4,4) to the storage block 3730 at the position (4, 3) now in an emptystate, and stores the pixel data therein. Simultaneously, the controlunit 3740 causes pixel data of the pixel block 3131 at the position (4,4) acquired at the timing T₂ to be stored in the storage block 3730 atthe corresponding position (4, 4).

At the timing T₃, the control unit 3740 moves the pixel data that isstored in a storage block 3730 at the position (5, 4) to a storage block3730 at the position (6, 4) in the outer circumferential direction, andstores the pixel data therein, and moves pixel data of the storage block3730 that corresponds to the pixel block 3131 at the position (4, 4) tothe storage block 3730 at the position (5, 4) now in an empty state, andstores the pixel data therein. Simultaneously, the control unit 3740causes pixel data of the pixel block 3131 at the position (4, 4)acquired at the timing T₃ to be stored in the storage block 3730 at thecorresponding position (4, 4).

At the timing T₄, the control unit 3740 moves the pixel data that isstored in a storage block 3730 at the position (4, 5) to a storage block3730 at the position (4, 6) in the outer circumferential direction, andstores the pixel data therein, and moves the pixel data of the storageblock 3730 that corresponds to the pixel block 3131 at the position (4,4) to the storage block 3730 at the position (4, 5) now in an emptystate, and stores the pixel data therein. Simultaneously, the controlunit 3740 causes pixel data of the pixel block 3131 at the position (4,4) acquired at the timing T₄ to be stored in the storage block 3730 atthe corresponding position (4, 4). At this time, the pixel data at thetiming from T₀ to T₄ are stored in the storage block 3730 at theposition (4, 4) that corresponds to the pixel block 3131 at the position(4, 4), and the storage blocks 3730 at the positions (3, 4), (4, 3), (5,4) and (4, 5) that surround the storage block 3730 two-dimensionally.

The control unit 3740 may move respective pieces of the pixel datastored in the storage blocks 3730 at the positions (3, 4), (4, 3), (5,4), (4, 5) to the storage blocks 3730 that are closest to the edges ofthe pixel area 3700 from among the adjacent storage blocks 3730. Thatis, the control unit 3740 may cause the respective pieces of the pixeldata stored in the storage blocks 3730 at the positions (3, 4), (4, 3),(5, 4), (4, 5) to the storage blocks 3730 at the positions (1, 4), (4,1), (6, 4) and (4, 6) that correspond to the edges of the pixel area3700, and store the pixel data therein.

At the timing T₅, the control unit 3740 passes the pixel data stored inall the storage blocks 3730 of the pixel area 3700 over to a memory oran arithmetic circuit in the subsequent step through a bus line. Thecontrol unit 3740 updates the cycle of frames, and repeats theabove-described operations from the timing T₀ to T₄.

The control unit 3740 fixes, at the reference frame rate, the frame rateof the pixel blocks 3131 along the outermost circumference of the pixelarea 3700 from among a plurality of the pixel blocks 3131. Meanwhile,because the adjacent storage blocks 3730 are limited when the high framerate pixel block 3131 is at the edge of the pixel area 3700, it is hardto distribute the pixel data two-dimensionally. Accordingly, the controlunit 3740 causes the high frame rate pixel block 3131 to be not at theoutermost circumference of the pixel area 3700. For example, the controlunit 3740 fixes, at the reference frame rate, the frame rate of thepixel blocks 3131 at the outermost circumference of the pixel area 3700.

Simultaneously, the control unit 3740 writes new pixel data in thestorage blocks 3730 that correspond respectively to all the pixel blocks3131, and transmits pixel data of the respective pixel blocks 3131collectively to an arithmetic processing circuit in the subsequent step.In this manner, the control unit 3740 can reduce the memory capacitybecause the storage blocks 3730 can be shared among a plurality of thepixel blocks 3131, by sequentially moving the pixel data of the highframe rate pixel block 3131 to the storage blocks 3730 that correspondto the adjacent pixel blocks 3131 in directions toward the edges of thepixel area 3700. The pixel data that is allocated to the plurality ofthe adjacent storage blocks 3730 may have, as header information,positional data of the pixel blocks 3131 that correspond to itselfwithin the pixel area 3700, and frame data that indicates a frame towhich it belongs, as the additional data.

Although in the present example, the control unit 3740 sequentiallymoves the pixel data of the high frame rate pixel block 3131 to thestorage blocks 3730 that correspond to adjacent pixel blocks 3131, andstores the pixel data therein, the control unit 3740 may move the pixeldata to every other storage blocks 3730, and may move the pixel data tostorage blocks 3730 in diagonal directions, instead of the row andcolumn directions, and store the pixel data therein. The control unit3740 may select storage blocks 3730 to which the pixel data is movedbased on frame rate information on each pixel block 3131.

FIG. 77 is a plan view that illustrates another configuration example ofthe imaging element 3100. In the present example, similar to the imagingelement 3100 illustrated in FIG. 76, pixel data is transmitted betweenstorage blocks 3730 that correspond to adjacent pixel blocks 3131. Notethat however, similar to the imaging element 3100 illustrated in FIG.75, the imaging element 3100 in the present example comprises thestorage units 3810 that are provided outside an area of the signalprocessing chip 3111 that overlaps with the pixel area 3700. The storageunit 3810 has storage areas 3820 that are divided by the number of thepixel blocks 3131 (in the present example, six) in the row direction,and storage areas 3822 that are divided by the number of the pixelblocks 3131 (in the present example, six) in the column direction. Thecontrol unit 3740 causes the pixel data that corresponds to the highframe rate pixel block 3131 to be stored in the predetermined storageareas 3820, 3822 in synchronization with the high frame rate.

The control unit 3740 may write the pixel data of the high frame ratepixel block 3131 at the position (4, 4) in the storage areas 3820, 3822that are associated with the low frame rate pixel blocks 3131 at theoutermost circumference in synchronization with the frame rate. Notethat the control unit 3740 may select storage areas 3820, 3822 that areassociated with pixel blocks 3131 not at the outermost circumferencebased on frame rate information on each pixel block 3131, and write thepixel data therein. The storage areas 3820, 3822 are shared by pixeldata of the high frame rate pixel blocks 3131 and pixel data of the lowframe rate pixel block 3131. In the present example, writing-in/readoutmay be performed for the respective storage areas 3820, 3822, and it isnot necessary to perform writing-in/readout for the respective storageblocks 3730 provided in the pixel block 3131; thus, the circuitconfiguration can be simplified. Also, the sizes of the respectivememory spaces of the storage areas 3820, 3822 in the storage unit 3810of the present example are the same. Furthermore, the positions of thememory spaces of the storage areas 3820, 3822 may be fixed within thestorage units 3810, or may be changed dynamically.

FIG. 78 illustrates a configuration and operations of a part of theimaging element 3100 according to another embodiment. The presentexample is different from the above-described embodiment in that thestorage unit 3114 is configured with a buffer memory having a multilayerstructure. The storage unit 3114 in the present example includes atemporary memory 3850 and a transfer memory 3860. The temporary memory3850 is a memory that has storage blocks 3830 that correspond to therespective pixel blocks 3131, and is used for controlling pixel data ofthe high data rate pixel block 3712. The transfer memory 3860 receivespixel data input from the temporary memory 3850, and transfers the pixeldata to a memory or an arithmetic circuit in the next step. The transfermemory 3860 has a storage area with at least the same size with thetotal storage area of the plurality of the storage blocks 3730. Here,the total storage area refers to the size of the memory space that thetemporary memory 3850 has. The temporary memory 3850 in the presentexample has functions and a configuration that are the same with thoseof the storage block 3730 illustrated in FIG. 76.

Here, it is assumed in the following example that the frame rate of thepixel block 3712 is five-fold of the reference frame rate. When thereference frame rate is 60 fps, the high frame rate is 300 fps. Theimaging timing at the high frame rate is as follows: At the clock timet=0, the timing is T₀; at the clock time t=1/300 s, the timing is T₁; atthe clock time t=2/300 s, the timing is T₂; at the clock time t=3/300 s,the timing is T₃; at the clock time t=4/300 s, the timing is T₄; and atthe clock time t=5/300 s, the timing is T₅.

The control unit 3740 causes all the pieces of pixel data of a subjectimaged at the timing T₀ to be stored in the storage blocks 3830 thatcorrespond respectively to all the pixel blocks 3131. The control unit3740 transfers the stored pixel data to the transfer memory 3860 at thetiming prior to T₁. That is, the control unit 3740 causes all the piecesof pixel data of a subject imaged at the timing T₀ to be copied to thecorresponding storage area 3870 of the transfer memory 3860 before nextpixel data is input from the pixel block 3712 that operates at a highframe rate, and stored therein.

At the timing T₁ that is synchronized with the high frame rate, thecontrol unit 3740 causes pixel data to be stored in the correspondingstorage block 3853 of the temporary memory 3850 from the high frame ratepixel block 3712 via the bus 3720. The control unit 3740 causes thepixel data stored in the storage block 3853 to be moved to an adjacentstorage block 3854 at the timing T₂ or timing prior to T₂, and storedtherein.

At the timing T₂, the control unit 3740 causes pixel data to be storedin the corresponding storage block 3853 of the temporary memory 3850from the pixel block 3712 via the bus 3720 in synchronization with thehigh frame rate.

The control unit 3740 causes the pixel data stored in the storage block3853 to be moved to an adjacent storage block 3855 at the timing T₃ ortiming prior to T₃, and stored therein. At the timing T₃, the controlunit 3740 causes pixel data to be stored in the corresponding storageblock 3853 of the temporary memory 3850 from the pixel block 3712 viathe bus 3720 in synchronization with the high frame rate. The controlunit 3740 causes the pixel data stored in the storage block 3853 to bemoved to an adjacent storage block 3856 at the timing T₄ or timing priorto T₄, and stored therein. At the timing T₄, the control unit 3740causes pixel data to be stored in the corresponding storage block 3853of the temporary memory 3850 from the pixel block 3712 via the bus 3720in synchronization with the high frame rate.

The control unit 3740 causes the pixel data stored in the storage blocks3854, 3855, 3856, 3857 of the temporary memory 3850 to be stored incorresponding storage areas 3864, 3865, 3866, 3867 of the transfermemory 3860 via a bus 3840, at the timing T₅ or timing prior to T₅. Thatis, after receiving pixel data at a high frame rate, among high framerates, which is immediately before the reference timing, the temporarymemory 3850 transfers the pixel data to the transfer memory 3860 untilnext pixel data at the reference timing is received.

Note that the control unit 3740 may further move pixel data stored inthe storage blocks 3854, 3855, 3856, 3857 adjacent to the storage block3853 to other adjacent storage blocks in synchronization with a highframe rate. The control unit 3740 transfers all the pieces of pixel datastored in the transfer memory 3860 to a memory or an arithmetic circuitin the subsequent step.

According to the present embodiment, because the storage block 3853 thatcorresponds to the high frame rate pixel block 3712, and the respectivestorage blocks 3854, 3855, 3856, 3857 adjacent to the storage block 3853only has to be connected by the transmission path 3710, it is notnecessary to connect all the storage blocks by the transmission path3710. Accordingly, pixel data can be moved at high speed. Also,writing-in/readout can be performed at high speed because a cache memorysuch as a SRAM can be used as the temporary memory 3850. Furthermore,because the storage block 3830 is not shared in the temporary memory3850, the circuit configuration that is necessary for writing-in/readoutcan be simplified. Furthermore, shared storage areas in the transfermemory 3860 are only storage areas that are adjacent to the storage area3863 that corresponds to the high frame rate pixel block 3712.Accordingly, an interconnection to connect the storage areas 3863 is notnecessary in the transfer memory 3860. Also, although it is assumed inthe example that the temporary memory 3850 has a configuration of thestorage block 3730 illustrated in FIG. 76, the temporary memory 3850 mayhave a configuration of any of the storage blocks 3730 illustrated inFIGS. 74 to 77.

FIG. 79 is a flowchart that illustrates another example of operations ofthe imaging device to generate and record a motion image. Operations ofFIG. 79 that are the same with those of FIG. 67 are given the samereference numbers, and explanation thereof is omitted.

In the operations of FIG. 79, in addition to or instead of the framerates in FIG. 67, thinning rates are made different between theattention area 3172 and the peripheral area 3176. More specifically, atStep S3120, the drive unit 3502 causes the pixel blocks 3131 included inthe attention area 3172 to execute electrical charge accumulation andimage signal output of pixels that are thinned at a low thinning rate,and causes the pixel blocks 3131 included in the peripheral area 3176 toexecute electrical charge accumulation and image signal output of pixelsthat are thinned at a high thinning rate. For example, pixels in thepixel blocks 3131 included in the attention area 3172 that are thinnedat the thinning rate of 0, that is, all the pixels are read out, andpixels in the pixel blocks 3131 included in the peripheral area 3176that are thinned at the thinning rate of 0.5, that is, a half of thepixels are read out.

In this case, the drive unit 3502 obtains image signals at differentthinning rates by separately driving a set of the reset transistors3303, the transfer transistors 3302, and the selecting transistors 3305of the pixel blocks 3131 included in the peripheral area 3176, and a setof the reset transistors 3303, the transfer transistors 3302, and theselecting transistors 3305 of the pixel blocks 3131 included in theattention area 3172.

At Step S3110, the motion image generating unit 3154 generates anattention area motion image that corresponds to the attention area 3172based on image signals of the attention area 3172 output at a lowthinning rate. The motion image generating unit 3154 similarly generatesa peripheral area motion image that corresponds to the peripheral area3176 based on the image signals of the peripheral area 3176 output at ahigh thinning rate. Also at Step S3112, the motion image generating unit3154 records the attention area motion image and the peripheral areamotion image, with information on the respective thinning rates beingadded thereto, in the recording unit 3505.

FIG. 80 illustrates an example of pixels 3188 to be read out at thethinning rate of 0.5 in one pixel block. In the example illustrated inFIG. 80, when a pixel block 3132 in the peripheral area 3176 is a Bayerarray, the pixels 3188 to be read out and pixels not to be read out areset for every other Bayer array, that is, every two pixels alternatelyin the vertical direction. Thereby, thinned readout can be performedwithout losing a color balance.

FIG. 81 is a flowchart that illustrates operations, corresponding toFIG. 79, of the imaging device to reproduce and display a motion image.Operations of FIG. 81 that are the same with those of FIG. 73 are giventhe same reference numbers, and explanation thereof is omitted.

At Step S3170 in FIG. 81, the motion image synthesizing unit 3156complements pixels of a frame of the peripheral area motion image tomatch its resolution with the resolution of a frame of the attentionarea motion image, and thereafter fits the frame of the attention areamotion image to the frame of the peripheral area motion image; thereby,a synthesized frame of the displayed image is formed. Thereby, imagesignals can be obtained at a high resolution from the attention area3172 including the main subject 3171, and also the data amount can bereduced by keeping the resolution of the peripheral area 3176 low.Accordingly, as compared with high speed readout from all the pixels,loads of driving and image processing can be reduced, and powerconsumption and heat generation can be suppressed.

Note that although the attention area 3172 is a rectangle in theexamples illustrated in FIGS. 61 to 81, the shape of the attention area3172 is not limited thereto. The attention area 3172 may be a convex orconcave polygon, or may have a doughnut shape with the peripheral area3176 positioned inside thereof or another shape as long as the attentionarea 3172 conforms to the boundary line of the pixel blocks 3131. Also,a plurality of the attention areas 3172 that are spaced apart from eachother may be set. In such a case, mutually different frame rates may beset for the attention areas 3172.

Also, frame rates of the attention area 3172 and the peripheral area3176 may be variable. For example, the moving amount of the main subject3171 may be detected with the elapse of a unit time, and a higher framerate may be set for the attention area 3172 if the moving amount of themain subject 3171 is larger. Also, selection of pixel blocks 3131 thatshould be included in the attention area 3172 may be updated at any timeduring the unit time, by tracking the main subject 3171.

Although motion image generation in FIGS. 67 and 79 starts when a userpresses down a record button, and motion image reproduction in FIGS. 73and 81 starts when a user presses down a reproduction button, thestarting time points are not limited thereto. As another example,triggered by a single button operation by a user, an operation of motionimage generation and an operation of motion image reproduction may becontinuously executed, and a through-image display (or also called alive view display) may be performed on the display unit 3506. In thiscase, a display for causing the user to recognize the attention area3172 may be superimposed. For example, a frame may be displayed over theboundary of the attention area 3172 on the display unit 3506, or theluminance of the peripheral area 3176 may be lowered or the luminance ofthe attention area 3172 may be raised.

In the operations in FIG. 79, thinning rates are made different betweenthe attention area 3172 and the peripheral area 3176. Instead of makingthe thinning rates different, the numbers of adjacent rows of pixelswhose pixel signals are added may be made different. For example, in theattention area 3172, the number of rows is one, which means that pixelsignals are output without addition among adjacent rows, and in theperipheral area 3176, the number of rows is larger than that for theattention area 3172, that is, for example two, which means that pixelsignals of pixels of two adjacent rows that are in the same columns areoutput. Thereby, similar to FIG. 79, the overall signal amount can bereduced while keeping the resolution of the attention area 3172 higherthan that of the peripheral area 3176.

Note that the motion image synthesizing unit 3156 may be provided in anexternal display apparatus, for example a PC, instead of being providedin the image processing unit 3511 of the imaging device 3500. Also, theabove-described embodiment may be applied not only to motion imagegeneration, but also to still image generation.

Also, although in the above-described embodiments, a plurality of thepixel blocks 3131 is divided into two areas, the attention area 3172 andthe peripheral area 3176, the number of division is not limited thereto,and the pixel blocks 3131 may be divided into three or more areas. Inthis case, pixel blocks 3131 that correspond to the boundary between theattention area 3172 and the peripheral area 3176 may be handled as aboundary area, and the boundary area may be controlled by using anintermediate value between a value of a control parameter used for theattention area 3172 and a value of a control parameter used for theperipheral area 3176. Thereby, it is possible to prevent the boundarybetween the attention area 3172 and the peripheral area 3176 fromlooking unnatural.

Accumulation periods and numbers of times of accumulation of electricalcharges, and the like may be made different between the attention area3172 and the peripheral area 3176. In this case, the attention area 3172and the peripheral area 3176 may be divided based on luminance, andfurthermore an intermediate area may be provided.

FIGS. 82A and 82B are diagrams for explaining an example of a scene andarea division. FIG. 82A illustrates a scene captured by a pixel area ofthe imaging chip 3113. Specifically, the scene includes simultaneously ashadowed subject 3601 and an intermediate subject 3602 included in anindoor environment, and a highlighted subject 3603 of an outdoorenvironment observed within a window frame 3604. When imaging, with aconventional imaging element, such a scene in which the contrast betweena highlighted portion and a shadowed portion is high, blocked-up shadowsoccur at the shadowed portion if electrical charge accumulation isexecuted by using the highlighted portion as a reference, and blown-outhighlights occur at the highlighted portion if electrical chargeaccumulation is executed by using the shadowed portion as a reference.That is, it can be said that, for a high contrast scene, the photo diodedoes not have a sufficient dynamic range that is needed for imagesignals to be output by one-time electrical charge accumulation that isuniform for the highlighted portion and the shadowed portion. To copewith this, in the present embodiment, a scene is divided into partialareas such as a highlighted portion and a shadowed portion, andsubstantial expansion of a dynamic range is attempted by making thenumbers of times of electrical charge accumulation mutually differentbetween photo diodes that correspond to respective areas.

FIG. 82B illustrates area division of a pixel area in the imaging chip3113. The computing unit 3512 analyzes the scene of FIG. 82A captured bythe photometry unit 3503 to divide the pixel area based on luminance.For example, the system control unit 3501 causes the photometry unit3503 to execute scene acquisition multiple times while changing exposureperiods, and the computing unit 3512 decides division lines of the pixelarea by referring to changes in distribution of blown-out highlightareas and blocked-up shadowed areas. In the example of FIG. 82B, thecomputing unit 3512 performs division into three areas, a shadowed area3611, an intermediate area 3612, and a highlighted area 3613.

The division line is defined along boundaries of pixel blocks 3131. Thatis, each divided area includes an integer number of groups. Then, pixelsof each group included in the same area perform electrical chargeaccumulation and pixel signal output the same number of times in aperiod that corresponds to a shutter speed decided by the computing unit3512. If pixels belong to different areas, electrical chargeaccumulation and pixel signal output are performed different numbers oftimes.

FIG. 83 is a diagram for explaining electrical charge accumulationcontrol for the respective areas divided in the example in FIGS. 82A and82B. Upon receiving an imaging stand-by instruction from a user, thecomputing unit 3512 decides a shutter speed T₀ based on an output fromthe photometry unit 3503. Furthermore, the computing unit 3512 performsdivision into the shadowed area 3611, the intermediate area 3612, andthe highlighted area 3613 in a manner as above-described, and decidesthe numbers of times of electrical charge accumulation based onrespective pieces of luminance information. The numbers of times ofelectrical charge accumulation are decided such that pixels are notsaturated by one-time electrical charge accumulation. For example, thenumbers of times of electrical charge accumulation are decided such that80 to 90% of accumulatable electrical charges is accumulated in aone-time electrical charge accumulation operation.

Here, electrical charge accumulation is performed once for the shadowedarea 3611. That is, the decided shutter speed T₀ and the electricalcharge accumulation period are caused to match. Also, electrical chargeaccumulation is performed twice for the intermediate area 3612. That is,a one-time electrical charge accumulation period is set to T₀/2, andelectrical charge accumulation is repeated twice during the shutterspeed T₀. Also, electrical charge accumulation is performed four timesfor the highlighted area 3613. That is, a one-time electrical chargeaccumulation period is set to T₀/4, and electrical charge accumulationis repeated four times during the shutter speed T₀.

Upon receiving an imaging instruction from a user at a clock time t=0,the drive unit 3502 applies reset pulses and transfer pulses to pixelsin groups belonging to the respective areas. This application triggers astart of electrical charge accumulation of all the pixels.

At a clock time t=T₀/4, the drive unit 3502 applies transfer pulses topixels in groups belonging to the highlighted area 3613. Then, the driveunit 3502 sequentially applies selection pulses to pixels in each groupto cause their respective pixel signals to be output to the outputinterconnection 3309. After pixel signals of all the pixels in thegroups are output, the drive unit 3502 applies reset pulses and transferpulses again to pixels in groups belonging to the highlighted area 3613to cause second electrical charge accumulation to be started.

Note that because selective output of pixel signals takes time, a timelag occurs between the end of first electrical charge accumulation andthe start of second electrical charge accumulation. When this time lagis substantially negligible, a one-time electrical charge accumulationperiod may be calculated by dividing the shutter speed T₀ by the numbersof times of electrical charge accumulation as described above. On theother hand, if not negligible, the shutter speed T₀ may be adjusted byconsidering the time, or the a one-time electrical charge accumulationperiod may be made shorter than the time obtained by dividing theshutter speed T₀ by the numbers of times of electrical chargeaccumulation.

At a clock time t=T₀/2, the drive unit 3502 applies transfer pulses topixels in groups belonging to the intermediate area 3612 and thehighlighted area 3613. Then, the drive unit 3502 sequentially appliesselection pulses to pixels in each group to cause their respective pixelsignals to be output to the output interconnection 3309. After pixelsignals of all the pixels in the groups are output, the drive unit 3502applies reset pulses and transfer pulses again to pixels in groupsbelonging to the intermediate area 3612 and the highlighted area 3613 tocause second electrical charge accumulation to be started for theintermediate area 3612 and cause third electrical charge accumulation tobe started for the highlighted area 3613.

At a clock time t=3T₀/4, the drive unit 3502 applies transfer pulses topixels in groups belonging to the highlighted area 3613. Then, the driveunit 3502 sequentially applies selection pulses to pixels in each groupto cause their respective pixel signals to be output to the outputinterconnection 3309. After pixel signals of all the pixels in thegroups are output, the drive unit 3502 applies reset pulses and transferpulses again to pixels in groups belonging to the highlighted area 3613to cause fourth electrical charge accumulation to be started.

At the clock time t=T₀, the drive unit 3502 applies transfer pulses topixels of all the areas. Then, the drive unit 3502 sequentially appliesselection pulses to pixels in each group to cause their respective pixelsignals to be output to the output interconnection 3309. According tothe above-described control, pixel signals that correspond to once arestored in each pixel memory 3414 that corresponds to the shadowed area3611, pixel signals that correspond to twice are stored in each pixelmemory 3414 that corresponds to the intermediate area 3612, and pixelsignals that correspond to four times are stored in each pixel memory3414 that corresponds to the highlighted area 3613.

Note that the drive unit 3502 may sequentially apply reset pulses andtransfer pulses to pixels in groups belonging to any area, andsequentially reset pixels in the groups belonging to the area. Triggeredby this application, pixels of each group may sequentially startelectrical charge accumulation. After the end of electrical chargeaccumulation of pixels in groups belonging to all the areas, the driveunit 3502 may apply transfer pulses to pixels in all the areas. Then,the drive unit 3502 may sequentially applies selection pulses to pixelsin each group to cause their respective pixel signals to be output tothe output interconnection 3309.

These pixel signals are sequentially transferred to the image processingunit 3511. The image processing unit 3511 generates image data with ahigh dynamic range based on the pixel signals. Specific processing isdescribed below.

FIG. 84 is a table that indicates a relationship between the number oftimes of integration and the dynamic range. Pixel data that correspondsto multiple times of repeatedly executed electrical charge accumulationare subjected to an integration process by the image processing unit3511 to form a part of image data with a high dynamic range.

When compared with, as a reference, a dynamic range of an area whosenumber of times of integration is once, that is, for which electricalcharge accumulation is performed once, a dynamic range of an area whosenumber of times of integration is twice, that is, whose output signal isintegrated by performing electrical charge accumulation twice isexpanded by one step. Similarly, when the number of times of integrationis four times, the dynamic range is expanded by two steps, and when thenumber of times of integration is 128, the dynamic range is expanded byseven steps. That is, in order to attempt to obtain n-steps of dynamicrange expansion, output signals may be integrated 2^(n) times.

Here, in order for the image processing unit 3511 to identify how manytimes electrical charge accumulation has been performed for whichdivided area, a 3-bit exponent indicating the number of times ofintegration is added to an image signal. As illustrated, exponents areallocated sequentially, 000 to the number of times of integration once,001 to twice, . . . , 111 to 128 times.

The image processing unit 3511 refers to an exponent of each piece ofpixel data received from the arithmetic circuit 3415 and when a resultof the reference shows that the number of times of integration is two ormore, executes an integration process of the pixel data. For example,when the number of times of integration is two (one step), upper 11 bitsof two pieces of 12-bit pixel data corresponding to electrical chargeaccumulation are added together to generate a single piece of 12-bitpixel data. Similarly, when the number of times of integration is 128(seven steps), upper 5 bits of 128 pieces of 12-bit pixel datacorresponding to electrical charge accumulation are added together togenerate a single piece of 12-bit pixel data. That is, upper bits, thenumber of which is obtained by subtracting, from 12, the number of stepscorresponding to the number of times of integration, are added togetherto generate a single piece of 12-bit pixel data. Note that lower bitsthat are not to be added are eliminated.

By performing processing in this manner, the luminance range thatprovides a gradation can be shifted to the high luminance side inaccordance with the number of times of integration. That is, 12 bits areallocated to a limited range on the high luminance side. Accordingly, agradation can be provided to an image area that conventionally includedblown-out highlights.

Note that however that, because 12 bits are allocated to differentluminance ranges of other divided areas, image data cannot be generatedby synthesis of simply connecting the areas. To cope with this, theimage processing unit 3511 performs a re-quantization process by using,as a reference, a highest luminance pixel and a lowest luminance pixelin order to make all the areas 12-bit image data while preservingobtained gradations as much as possible. Specifically, quantization isexecuted by performing gamma conversion so that the smoother gradationscan be preserved. By performing processing in this manner, image datawith a high dynamic range can be obtained.

Note that the description of the number of times of integration is notlimited to a 3-bit exponent being added to pixel data asabove-described, but the number of times of integration may be describedas accompanying information other than the pixel data. Also, theexponent may be omitted from pixel data, and instead the number of timesof integration may be acquired at the time of an adding process bycounting the number of pieces of pixel data stored in the pixel memory3414.

Also, although in the above-described image processing, are-quantization process to make all the areas 12-bit image data isexecuted, the number of output bits may be increased from the bit numberof pixel data, in accordance with an upper limit number of times ofintegration. For example, if the upper limit number of times ofintegration is defined as 16 (four steps), all the areas may be made,for 12-bit pixel data, 16-bit image data. By performing processing inthis manner, image data can be generated without cancellation of digits.

Next, a series of imaging operation processes is explained. FIG. 85 is aflow diagram showing processing of imaging operations. The flow startswhen a power supply of the imaging device 3500 is turned on.

At Step S3201, the system control unit 3501 waits for a switch SW1 to bepressed down, which is an imaging stand-by instruction. When pressingdown of the switch SW1 is sensed, the flow proceeds to Step S3202.

At Step S3202, the system control unit 3501 executes photometryprocessing. Specifically, upon obtaining an output of the photometryunit 3503, the computing unit 3512 calculates luminance distribution ofa scene. Then, the flow proceeds to Step S3203, and as described above,a shutter speed, area division, the number of times of integration, andthe like are decided.

Upon completion of the imaging stand-by operation, the flow proceeds toStep S3204, and waits for a switch SW2 to be pressed down, which is animaging instruction. At this time, when the elapsed time exceeds apredetermined time Tw (YES at Step S3205), the flow returns to StepS3201. When pressing down of the switch SW2 is sensed before the elapsedtime exceeds the time Tw (NO at Step S3205), the flow proceeds to StepS3206.

At Step S3206, the drive unit 3502 that has received an instruction ofthe system control unit 3501 executes an electrical charge accumulationprocess and a signal readout process that are explained by using FIG.83. Then, upon completion of entire signal readout, the flow proceeds toStep S3207, the image processing explained by using FIG. 84 is executed,and a recording process of recording generated image data in therecording unit is executed.

Upon completion of the recording process, the flow proceeds to StepS3208, and it is determined whether the power supply of the imagingdevice 3500 has been turned off. When the power supply has not beenturned off, the flow returns to Step S3201, and when the power supplyhas been turned off, the series of imaging operation processes ends.

FIG. 86 is a block diagram that illustrates a specific configuration ofthe signal processing chip 3111 as one example. The areas in the figurethat are surrounded by dotted lines indicate the pixel data processingunit 3910 that is provided to each pixel block 3131.

The signal processing chip 3111 serves functions of the drive unit 3502.The signal processing chip 3111 includes a sensor control unit 3441, ablock control unit 3442, a synchronization control unit 3443, and asignal control unit 3444 that serve divided control functions, and adrive control unit 3420 that performs overall control on the respectivecontrol units. The drive control unit 3420 converts instructions fromthe system control unit 3501 into control signals that can be executedby the respective control units, and passes them over to the respectivecontrol units.

The sensor control unit 3441 performs transmission control on controlpulses that are to be transmitted to the imaging chip 3113 and relate toelectrical charge accumulation and electrical charge readout of eachpixel. Specifically, the sensor control unit 3441 controls the start andend of electrical charge accumulation by transmitting reset pulses andtransfer pulses to target pixels, and causes pixel signals to be outputto the output interconnection 3309 by transmitting selection pulses toreadout pixels.

The block control unit 3442 executes transmission of specifying pulsesthat are to be transmitted to the imaging chip 3113 and specify a pixelblock 3131 to be controlled. As explained by using FIG. 82B, etc.,divided areas may include a plurality of mutually adjacent pixel blocks3131. Pixel blocks 3131 belonging to the same area form a single block.Pixels that are included in the same block start electrical chargeaccumulation at the same timing, and end the electrical chargeaccumulation at the same timing. To cope with this, the block controlunit 3442 plays a role of forming blocks of pixel blocks 3131 bytransmitting specifying pulses to pixel blocks 3131 to be targets basedon designation by the drive control unit 3420. Transfer pulses and resetpulses that each pixel receives via the TX interconnection 3307 and thereset interconnection 3306 are logical AND of each pulse transmitted bythe sensor control unit 3441 and specifying pulses transmitted by theblock control unit 3442.

In this manner, by controlling each area as a mutually independentblock, the electrical charge accumulation control explained by usingFIG. 83 can be realized. The block-formation designation by the drivecontrol unit is described in detail below. Note that pixels included inthe same block may not start electrical charge accumulation at the sametiming. That is, the drive control unit 3420 may apply reset pulses andtransfer pulses to pixels included in the same block at differenttiming. Also, after terminating electrical charge accumulation of pixelsincluded in the same block after the same accumulation period, the drivecontrol unit 3420 may sequentially apply selection pulses to the pixelsin the block, and sequentially read out their respective pixel signals.

The synchronization control unit 3443 transmits a synchronization signalto the imaging chip 3113. Each pulse becomes active in the imaging chip3113 in synchronization with the synchronization signal. For example, byadjusting the synchronization signal, random control, thinning control,and the like only on particular pixels among pixels belonging to thesame pixel block 3131 can be realized.

The signal control unit 3444 mainly performs timing control on the A/Dconverter 3412. Pixel signals output via the output interconnection 3309are input to the A/D converter 3412 through a CDS circuit 3410 and themultiplexer 3411. The A/D converter 3412 is controlled by the signalcontrol unit 3444 to convert the input pixel signals into digital pixeldata. The pixel data converted into digital signals is passed over tothe de-multiplexer 3413, and is stored as a pixel value of digital datain the pixel memory 3414 corresponding to each pixel. The pixel memory3414 is one example of the storage block 3730.

The signal processing chip 3111 has a timing memory 3430, as anaccumulation control memory, that stores block division informationabout which pixel blocks 3131 are to be combined to form a block, andinformation on the number of times of accumulation about how many timeseach block formed repeats electrical charge accumulation. The timingmemory 3430 is configured for example with a flash RAM.

As described above, which pixel blocks 3131 are to be combined to form ablock is decided by the system control unit 3501 based on a detectionresult of luminance distribution detection of a scene that is executedprior to a series of imaging sequence. The decided blocks are dividedfor example into a first block, a second block, . . . , and defined bywhich pixel blocks 3131 are included therein. The drive control unit3420 receives the block division information from the system controlunit 3501, and stores it in the timing memory 3430.

Also, the system control unit 3501 decides how many times each blockrepeats electrical charge accumulation based on a detection result ofluminance distribution. The drive control unit 3420 receives theinformation on the number of times of accumulation from the systemcontrol unit 3501, and stores it in the timing memory 3430 by pairingthe information on the number of times of accumulation with thecorresponding block division information. By storing the block divisioninformation and the information on the number of times of accumulationin the timing memory 3430 in this manner, the drive control unit 3420may execute a series of electrical charge accumulation controlindependently by successively referring to the timing memory 3430. Thatis, when controlling acquisition of a single image, once the drivecontrol unit 3420 receives a signal of an imaging instruction from thesystem control unit 3501, the drive control unit 3420 thereafter is ableto complete accumulation control without receiving an instruction aboutcontrol on each pixel from the system control unit 3501 each time.

The drive control unit 3420 receives, from the system control unit 3501,block division information and information on the number of times ofaccumulation that are updated based on results of photometry (detectionresults of luminance distribution) executed in synchronization with animaging stand-by instruction, and as appropriate updates stored contentsof the timing memory 3430. For example, the drive control unit 3420updates the timing memory 3430 in synchronization with an imagingstand-by instruction or an imaging instruction. With this configuration,faster electrical charge accumulation control is realized, and thesystem control unit 3501 may execute other processing in parallel withelectrical charge accumulation control executed by the drive controlunit 3420.

The drive control unit 3420 which executes electrical chargeaccumulation control on the imaging chip 3113 further refers to thetiming memory 3430 in execution of readout control. For example, thedrive control unit 3420 refers to information on the number of times ofaccumulation of each block to store pixel data output from thede-multiplexer 3413 in a corresponding address of the pixel memory 3414.

The drive control unit 3420 reads out target pixel data of each pixelblock from the pixel memory 3414 according to a delivery request fromthe system control unit 3501, and passes it over to the image processingunit 3511. At this time, the drive control unit 3420 passes theadditional data corresponding to the respective pieces of target pixeldata together over to the image processing unit 3511. The pixel memory3414 has a memory space that can store pixel data corresponding to themaximum number of times of integration about each pixel block asdescribed above, and stores, as pixel values, respective pieces of pixeldata corresponding to the number of times of accumulation executed. Forexample, because when electrical charge accumulation is repeated fourtimes in a block, pixels included in the block output pixel signals thatcorrespond to the four times, a memory space in the pixel memory 3414for each pixel stores four pixel values. When having received, from thesystem control unit 3501, a delivery request that requests pixel data ofa particular pixel, the drive control unit 3420 specifies an address ofthe particular pixel on the pixel memory 3414, reads out all the piecesof stored pixel data, and passes them over to the image processing unit3511. For example when four pixel values are stored, all the four pixelvalues are sequentially passed over, and when only one pixel value isstored, the pixel value is passed over.

The drive control unit 3420 can read out pixel data stored in the pixelmemory 3414, pass it to the arithmetic circuit 3415, and cause thearithmetic circuit 3415 to execute the above-described integrationprocess. The pixel data having been subjected to the integration processis stored in a target pixel address of the pixel memory 3414. The targetpixel address may be provided adjacent to an address space before theintegration process, or may be the same address so that pixel data iswritten over the pixel data before the integration process. Also, adedicated space that collectively stores pixel values of respectivepixels after the integration process may be provided. When havingreceived, from the system control unit 3501, a delivery request thatrequests pixel data of a particular pixel, the drive control unit 3420can pass the pixel data after the integration process over to the imageprocessing unit 3511 depending on the form of the delivery request. Ofcourse, pieces of pixel data before and after the integration processmay passed over together.

A data transfer interface that transmits pixel data according to adelivery request is provided to the pixel memory 3414. The data transferinterface is connected with a data transfer line that connects with theimage processing unit 3511. The data transfer line 3920 is configuredfor example with a serial bus. In this case, a delivery request from thesystem control unit 3501 to the drive control unit 3420 is executed byaddressing that utilizes an address bus.

Transmission of pixel data by the data transfer interface is not limitedto an addressing system, but may adopt various systems. For example, atthe time of data transfer, a double data rate system in which bothrising and falling of a clock signal used for synchronization of eachcircuit are utilized to perform processing may be adopted. Also, a bursttransfer system of transferring data at once by partially omittingprocedures such as addressing, and attempting speed up may be adopted.Also, a bus system of using lines that connect a control unit, a memoryunit, and an input/output unit in parallel, and a serial system oftransferring data in series on a bit by bit basis may be adopted incombination.

With this configuration, because the image processing unit 3511 canreceive only necessary pieces of pixel data, the image processing unit3511 can complete image processing at high speed particularly whenforming a low resolution image. Also, because when the arithmeticcircuit 3415 is caused to execute the integration process, the imageprocessing unit 3511 does not have to execute the integration process,speeding up of the image processing may be attempted by functionaldivision and parallel processing.

By using the signal processing chip 3111 in FIG. 86, image processingmay be performed after acquiring pixel data by using different controlparameters between the attention area 3172 and the peripheral area 3176.For example, although in FIGS. 67 to 70, a motion image is generatedfrom images that are acquired at frame rates that are different betweenthe attention area 3172 and the peripheral area 3176, instead of this,an S/N ratio may be improved by performing image processing of averagingimages acquired at a high frame rate. In this case, the drive controlunit 3420 obtains pixel signals that correspond to multiple times, forexample four times, from the attention area 3172 for example whileobtaining pixel signals that corresponds to once from the peripheralarea 3176, and stores the pixel data in the pixel memory 3414. Thearithmetic circuit 3415 reads out a plurality of pieces of pixel dataobtained, from the pixel memory 3414, for each pixel of the attentionarea 3172, and averages them for respective pixels. Thereby, randomnoises of each pixel of the attention area 3172 are reduced, and an S/Nratio of the attention area 3172 can be improved.

Note that a memory 3930 may be connected with the data transfer line3920. The memory 3930 may be a volatile memory that sequentially storespixel data from the pixel memory 3414 at designated addresses. Forexample, the memory 3930 is a DRAM. The rate of transferring pixel datafrom the pixel memory 3414 to the memory 3930 may be the same or slowerthan the reference frame rate. The memory 3930 functions as a buffer fordata transmission from the pixel memory 3414 to the image processingunit 3511. That is, the memory 3930 buffers at least a part of pixeldata output by the pixel memory 3414 when the rate of transferring datafrom the plurality of pixel memories 3414 is faster than the dataprocessing rate in the image processing unit 3511. For example, thememory 3930 stores pixel data of each reference frame rate, and pixeldata of the pixel block 3131 that operates at the high frame rate fromthe pixel memory 3414.

While the embodiment(s) of the present invention has (have) beendescribed, the technical scope of the invention is not limited to theabove described embodiment(s). It is apparent to persons skilled in theart that various alterations and improvements can be added to theabove-described embodiment(s). It is also apparent from the scope of theclaims that the embodiments added with such alterations or improvementscan be included in the technical scope of the invention.

The operations, procedures, steps, and stages of each process performedby an apparatus, system, program, and method shown in the claims,embodiments, or diagrams can be performed in any order as long as theorder is not indicated by “prior to,” “before,” or the like and as longas the output from a previous process is not used in a later process.Even if the process flow is described using phrases such as “first” or“next” in the claims, embodiments, or diagrams, it does not necessarilymean that the process must be performed in this order.

What is claimed is:
 1. An imaging element comprising: an imaging unitthat has: a plurality of groups each including at least one pixel; and aplurality of signal readout units that are each provided to each of thegroups and read out a signal from the pixel; and a control unit thatcontrols the signal readout unit in at least one group among theplurality of groups.
 2. The imaging element according to claim 1,wherein each of the plurality of groups includes a plurality of thepixels.
 3. The imaging element according to claim 1, wherein the controlunit selects at least one group among the plurality of groups andcontrols the signal readout unit by using a control parameter that isdifferent from a control parameter that is used for another group amongthe plurality of groups.
 4. The imaging element according to claim 3,wherein the control parameter includes a frame rate, and the controlunit controls the signal readout unit that corresponds to the at leastone group at a first frame rate, and controls the signal readout unitthat corresponds to the other group at a second frame rate that isdifferent from the first frame rate.
 5. An imaging device comprising:the imaging element according to claim 4; and a motion image generatingunit that generates a motion image of a first partial area thatcorresponds to the at least one group based on a signal of the at leastone group output at the first frame rate, and generates a motion imageof a second partial area that corresponds to the other group based on asignal of the other group output at the second frame rate.
 6. Theimaging device according to claim 5, wherein the motion image generatingunit stores, in a storage unit, area information and timing informationin association with the motion image of the first partial area and themotion image of the second partial area, the area information indicatinga range of the first partial area in relation to an entire area that isimaged by the imaging unit, the timing information indicating arelationship between output timing of the signal output at the firstframe rate and the signal output at the second frame rate.
 7. Theimaging element according to claim 3, wherein the control parameterincludes a thinning rate, and the control unit controls the signalreadout unit corresponding to the at least one group at a first thinningrate, and controls the signal readout unit that corresponds to the othergroup at a second thinning rate that is different from the firstthinning rate.
 8. An imaging device comprising: the imaging elementaccording to claim 7; and a motion image generating unit that generatesa motion image of a first partial area that corresponds to the at leastone group based on a signal of the at least one group output at thefirst thinning rate, and generates a motion image of a second partialarea that corresponds to the other group based on a signal of the othergroup output at the second thinning rate.
 9. The imaging deviceaccording to claim 8, wherein the motion image generating unit stores,in a storage unit, area information and thinning information inassociation with the motion image of the first partial area and themotion image of the second partial area, the area information indicatinga range of the first partial area in relation to an entire area that isimaged by the imaging unit, the thinning information indicating arelationship between the first thinning rate and the second thinningrate.
 10. The imaging element according to claim 3, wherein the controlparameter includes a number of added rows or added columns thatindicates a number of rows or columns whose pixel signals are added, andthe control unit controls the signal readout unit that corresponds tothe at least one group so that signals of a first number of rows orcolumns are added, and controls the signal readout unit that correspondsto the other group so that signals of a second number of rows or columnsthat is different from the first number of rows or columns are added.11. The imaging element according to claim 3, wherein the controlparameter includes an electrical charge accumulation period, and thecontrol unit causes the at least one group to execute electrical chargeaccumulation multiple times to output respective signals in a periodduring which the other group is caused to execute electrical chargeaccumulation once.
 12. The imaging element according to claim 3, whereinthe control parameter includes a number of bits for digitization of apixel signal, and the control unit digitizes a signal of the at leastone group with a number of bits that is larger than a number of bits ofthe other group among the plurality of groups.
 13. The imaging elementaccording to claim 1, further comprising: a subject estimating unit thatestimates a main subject based on an image imaged by the imaging unit;and a group selecting unit that selects, as the at least one group, agroup on which image light of the main subject that is estimated by thesubject estimating unit is incident.
 14. The imaging element accordingto claim 3, wherein the control unit uses an intermediate value betweena value of the control parameter of the at least one group and a valueof the control parameter of the other group as the control parameter fora group, among the plurality of groups, that is present at a boundarybetween the at least one group and the other group.
 15. The imagingelement according to claim 1, wherein an imaging chip in which theplurality of groups are two-dimensionally disposed, and a signalprocessing chip in which at least a part of the control unit is disposedare layered.
 16. The imaging element according to claim 15, wherein theimaging chip is formed with a backside illuminating type CMOS chip. 17.An imaging element comprising: an imaging unit that has: a plurality ofgroups each including at least one pixel; and a plurality of signalreadout units that are each provided to each of the groups and read outa signal from the pixel; and a plurality of control units that are eachprovided to each of the groups, and controls the signal readout unitbased on a signal from the pixel.
 18. The imaging element according toclaim 17, wherein the each of the plurality of groups includes aplurality of the pixels.
 19. An imaging element comprising: an imagingunit having an imaging area in which a first pixel and a second pixelare provided, a first readout circuit that reads out a first pixelsignal output from the first pixel, and a second readout circuit thatreads out a second pixel signal output from the second pixel; a firstcomputing unit that computes a first evaluation value based on the firstpixel signal; a second computing unit that computes a second evaluationvalue based on the second pixel signal; a first control unit thatperforms control on exposure or readout of the first pixel based on thefirst evaluation value; and a second control unit that performs controlon exposure or readout of the second pixel based on the secondevaluation value.
 20. The imaging element according to claim 19, whereinthe imaging area includes a first area in which a plurality of the firstpixels is provided and a second area in which a plurality of the secondpixels is provided, the first control unit performs control on exposureor readout of the plurality of first pixels provided in the first area,and the second control unit performs control on exposure or readout ofthe plurality of second pixels provided in the second area.
 21. Theimaging element according to claim 19, wherein the first control unitperforms frame rate control on readout of the first pixel based on thefirst evaluation value, and the second control unit performs frame ratecontrol on readout of the second pixel based on the second evaluationvalue.
 22. The imaging element according to claim 20, wherein the firstcontrol unit controls a thinning rate for thinning and reading out aplurality of the first pixels provided in the first area based on thefirst evaluation value, and the second control unit controls a thinningrate for thinning and reading out a plurality of the second pixelsprovided in the second area based on the second evaluation value. 23.The imaging element according to claim 20, wherein the first controlunit controls a number of added pixels for adding and reading out aplurality of the first pixels provided in the first area based on thefirst evaluation value, and the second control unit controls a number ofadded pixels for adding and reading out a plurality of the second pixelsprovided in the second area based on the second evaluation value. 24.The imaging element according to claim 19, wherein the first computingunit computes the first evaluation value according to the second pixelsignal.
 25. The imaging element according to claim 19, wherein the firstcomputing unit computes the first evaluation value according to thesecond evaluation value.
 26. The imaging element according to claim 19,the imaging element being configured with: an imaging chip that has theimaging unit; and a signal processing chip that has the first computingunit and the second computing unit, and is bonded with the imaging chipby layering.
 27. The imaging element according to claim 26, wherein theimaging chip is formed with a backside illuminating type CMOS chip. 28.An imaging device comprising the imaging element according to claim 19.29. An imaging element comprising: an imaging unit that has: a pluralityof groups each including at least one pixel; and a plurality of signalreadout units that are each provided to each of the groups and read outa signal from the pixel; and a plurality of computing units that areprovided to each of the groups and transmit information about control onthe signal readout unit to an image processing unit that performs imageprocessing on the signal.
 30. The imaging element according to claim 1,wherein each of the plurality of groups includes a plurality of pixels.31. An imaging element comprising: an imaging unit having an imagingarea in which a first pixel and a second pixel are disposed, a firstreadout circuit that reads out a first pixel signal output from thefirst pixel, and a second readout circuit that reads out a second pixelsignal output from the second pixel; a first computing unit thatcomputes a first evaluation value based on the first pixel signal, andtransmits the computed first evaluation value to an image processingunit in a subsequent step that performs image processing on first pixeldata that corresponds to the first pixel signal; and a second computingunit that computes a second evaluation value based on the second pixelsignal, and transmits the computed second evaluation value to an imageprocessing unit in a subsequent step that performs image processing onsecond pixel data that corresponds to the second pixel signal.
 32. Theimaging element according to claim 31, wherein the imaging area includesa first pixel block in which a plurality of the first pixels aredisposed, and a second pixel block in which a plurality of the secondpixels are disposed, the first computing unit calculates the firstevaluation value based on a plurality of the first pixel signals outputby the plurality of first pixels included in the first pixel block, andthe second computing unit calculates the second evaluation value basedon a plurality of the second pixel signals output by the plurality ofsecond pixels included in the second pixel block.
 33. The imagingelement according to claim 31, wherein the first computing unitassociates the first evaluation value with the first pixel data, and thesecond computing unit associates the second evaluation value with thesecond pixel data.
 34. The imaging element according to claim 31,wherein the first computing unit provides a data code to the firstevaluation value, the data code indicating computational contents of thefirst evaluation value, and the second computing unit provides a datacode to the second evaluation value, the data code indicatingcomputational contents of the second evaluation value.
 35. The imagingelement according to claim 31, wherein the first computing unit computesthe first evaluation value based further on the second pixel signal. 36.The imaging element according to claim 31, wherein the first computingunit computes the first evaluation value based further on the secondevaluation value in the second computing unit or a computation result ina process in which the second computing unit computes the secondevaluation value.
 37. The imaging element according to claim 36, whereinthe first computing unit has: a corresponding block calculating unitthat performs predetermined computation on the first pixel data; and anaverage calculating unit that performs predetermined computation on thesecond evaluation value or the computation result in the secondcomputing unit, and a computation result in the corresponding blockcalculating unit; and the first computing unit transmits the computationresult in the corresponding block calculating unit to the secondcomputing unit, and transmits a computation result in the averagecalculating unit to the image processing unit as the first evaluationvalue.
 38. The imaging element according to claim 37, wherein the firstcomputing unit further has an average-average calculating unit thatcompresses the first pixel data based on the computation result in theaverage calculating unit.
 39. The imaging element according to claim 31,wherein at least one of the first computing unit and the secondcomputing unit performs predetermined computation on the pixel data in acurrent frame by using the pixel data in a previous frame.
 40. Theimaging element according to claim 32, further comprising: a first A/Dconverter that is provided corresponding to the first pixel block, andconverts the respective first pixel signal into the first pixel data;and a second A/D converter that is provided corresponding to the secondpixel block, and converts the respective second pixel signal into thesecond pixel data.
 41. The imaging element according to claim 31,wherein the imaging unit is formed in an imaging chip, and the firstcomputing unit and the second computing unit are formed in a signalprocessing chip that is layered with the imaging chip.
 42. The imagingelement according to claim 41, wherein the imaging chip is a backsideilluminating type CMOS chip.
 43. The imaging element according to claim31, wherein the image processing unit is provided in the imagingelement, and performs image processing on the first pixel data and thesecond pixel data based on the first evaluation value and the secondevaluation value.
 44. An imaging device comprising the imaging elementaccording to claim
 31. 45. The imaging device according to claim 44,wherein the image processing unit is provided outside the imagingelement, and performs image processing on the first pixel data and thesecond pixel data based on the first evaluation value and the secondevaluation value.
 46. An imaging element comprising: an imaging unitthat has a plurality of groups each including at least one pixel; and astorage unit that has a plurality of storage blocks that are providedcorresponding to the plurality of groups, and store a signal from apixel in the respectively corresponding group, and store a signal from apixel in a group other than the respectively corresponding group. 47.The imaging element according to claim 46, wherein each of the pluralityof groups includes a plurality of pixels.
 48. The imaging elementaccording to claim 46, wherein frame rates that indicate cycles ofgenerating the signals in the plurality of groups can be selected on agroup-by-group basis from at least two frame rates that are a referenceframe rate and a high frame rate that indicates a cycle shorter than thereference frame rate, and the imaging element further comprises acontrol unit that causes the signal of the storage block thatcorresponds to the group of the high frame rate to be stored in thestorage block that corresponds to the group of the reference frame rate.49. The imaging element according to claim 48, wherein the respectivegroups are divided into blocks each including a plurality of the groupsthat are distributed being spaced apart at certain intervals in row andcolumn directions over the entire pixel area, and the respective storageblocks that correspond to the groups in the block are shared by all thegroups in the block, and when the signal is already stored in thestorage block that corresponds to the group of the high frame rate, thecontrol unit causes the signal that corresponds to the group to bestored in any of the storage blocks in the same block with the group.50. The imaging element according to claim 46, wherein frame rates thatindicate cycles of generating the signals in the groups can be selectedon a group-by-group basis from at least two frame rates that are areference frame rate and a high frame rate that indicates a cycleshorter than the reference frame rate, and the imaging element furthercomprises a control unit that, when the signal is already stored in thestorage block that corresponds to the group of the high frame rate,moves the signal of the storage block to the storage block thatcorresponds to the group of the reference frame rate, and causes thesignal that corresponds to the group to be stored in the correspondingstorage block.
 51. The imaging element according to claim 50 furthercomprising a transmission path that transmits the signal between thestorage blocks that correspond to the adjacent groups, wherein thecontrol unit causes the signal that corresponds to the group of the highframe rate to be sequentially moved to the adjacent storage block insynchronization with the high frame rate.
 52. The imaging elementaccording to claim 51, wherein the control unit causes the respectivesignal to be moved to the storage block that is closest to an edge of apixel area among the adjacent storage blocks.
 53. The imaging elementaccording to claim 52, wherein the control unit fixes, at the referenceframe rate, the frame rate of the group along an outermost circumferenceof a pixel area among the plurality of groups.
 54. The imaging elementaccording to claim 48, wherein the respective storage blocks arememories provided to the respective groups.
 55. The imaging elementaccording to claim 46, wherein the storage unit further has a transfermemory which has a storage area with at least the same size with a totalstorage area of the plurality of storage blocks and to which the signalsstored in the plurality of storage blocks are transferred at apredetermined cycle.
 56. An imaging device comprising the imagingelement according to claim
 46. 57. The imaging element according toclaim 1, further comprising a plurality of A/D converters that are eachprovided to each of the plurality of groups, and convert a signal fromthe pixel into pixel data.
 58. The imaging element according to claim17, further comprising a plurality of A/D converters that are eachprovided to each of the plurality of groups, and convert a signal fromthe pixel into pixel data.
 59. The imaging element according to claim20, further comprising: a first A/D converter that is providedcorresponding to the first area, and converts the respective first pixelsignal into first pixel data; and a second A/D converter that isprovided corresponding to the second area, and converts the respectivesecond pixel signal into second pixel data.
 60. The imaging elementaccording to claim 29, further comprising a plurality of A/D convertersthat are each provided to each of the plurality of groups, and convert asignal from the pixel into pixel data.
 61. The imaging element accordingto claim 46, further comprising a plurality of A/D converters that areeach provided to each of the plurality of groups, and convert a signalfrom the pixel into pixel data.